2024-06-13T01:06:40.449 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-13T01:06:40.449 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577 --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/orig.config.yaml 2024-06-13T01:06:40.521 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-13T01:06:40.612 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577 branch: squid description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7752577' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 1373963c202323287783ccb099bbae1ec8f14a5d owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8815 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 250/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1373963c202323287783ccb099bbae1ec8f14a5d targets: smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA093QoEmfHJkn1GCACnAIFpbhEPytpFtx7vgVAqP/IGpqQOY27buDHkllIhlpf46rUN7cS2VPvU7CsEjm0QCaU= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTYnAV/jcuI1xUkbPKHpyJUety+6LqtPz3KLQFL9E8YEDbsJ6cIy575l3KwwmbZj4XNo4b/Dd2exuVW0jT9yw0= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /home/teuthology/src/git.ceph.com_ceph_squid/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-12_21:24:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-13T01:06:40.612 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa; will attempt to use it 2024-06-13T01:06:40.613 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks 2024-06-13T01:06:40.613 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-13T01:06:40.615 INFO:teuthology.task.internal:Checking packages... 2024-06-13T01:06:40.636 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-13T01:06:40.636 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:06:40.636 INFO:teuthology.packaging:ref: None 2024-06-13T01:06:40.636 INFO:teuthology.packaging:tag: None 2024-06-13T01:06:40.636 INFO:teuthology.packaging:branch: squid 2024-06-13T01:06:40.636 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:06:40.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-13T01:06:40.887 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2998.g1373963c 2024-06-13T01:06:40.889 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-13T01:06:40.895 INFO:teuthology.task.internal:no buildpackages task found 2024-06-13T01:06:40.895 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-13T01:06:40.900 INFO:teuthology.task.internal:Saving configuration 2024-06-13T01:06:40.918 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-13T01:06:40.925 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-13T01:06:40.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-13 01:01:29.266883', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA093QoEmfHJkn1GCACnAIFpbhEPytpFtx7vgVAqP/IGpqQOY27buDHkllIhlpf46rUN7cS2VPvU7CsEjm0QCaU='} 2024-06-13T01:06:40.974 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-13 01:01:29.265120', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTYnAV/jcuI1xUkbPKHpyJUety+6LqtPz3KLQFL9E8YEDbsJ6cIy575l3KwwmbZj4XNo4b/Dd2exuVW0jT9yw0='} 2024-06-13T01:06:40.974 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-13T01:06:40.980 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-13T01:06:40.980 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-13T01:06:40.980 INFO:teuthology.run_tasks:Running task console_log... 2024-06-13T01:06:41.035 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff300d47be0>, signals=[15]) 2024-06-13T01:06:41.035 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-13T01:06:41.041 INFO:teuthology.task.internal:Opening connections... 2024-06-13T01:06:41.041 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-06-13T01:06:41.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:06:41.115 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-06-13T01:06:41.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:06:41.192 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-13T01:06:41.198 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-06-13T01:06:41.216 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-06-13T01:06:41.216 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-06-13T01:06:41.272 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-06-13T01:06:41.272 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-13T01:06:41.273 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-13T01:06:41.274 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-06-13T01:06:41.299 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-06-13T01:06:41.317 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-06-13T01:06:41.317 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-06-13T01:06:41.373 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-13T01:06:41.374 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-06-13T01:06:41.375 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-13T01:06:41.375 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-13T01:06:41.375 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-13T01:06:41.375 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-06-13T01:06:41.451 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-13T01:06:41.460 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-13T01:06:41.466 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-13T01:06:41.466 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-06-13T01:06:41.468 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-06-13T01:06:41.484 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-13T01:06:41.490 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-13T01:06:41.491 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-06-13T01:06:41.525 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-06-13T01:06:41.618 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-13T01:06:41.679 INFO:teuthology.run_tasks:Running task kernel... 2024-06-13T01:06:41.693 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-13T01:06:41.693 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-13T01:06:41.693 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-13T01:06:41.694 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-13T01:06:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:06:41.710 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-06-13T01:06:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:06:41.711 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-06-13T01:06:41.766 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-437.el9.x86_64 2024-06-13T01:06:41.767 INFO:teuthology.task.kernel:Running kernel on smithi132: 5.14.0-437.el9.x86_64 2024-06-13T01:06:41.767 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2024-06-13T01:06:41.768 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-437.el9.x86_64 2024-06-13T01:06:41.768 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-437.el9.x86_64 2024-06-13T01:06:41.769 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-06-13T01:06:45.334 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.1 MB 00:01 2024-06-13T01:06:45.513 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.1 MB 00:02 2024-06-13T01:06:50.048 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 20 MB 00:03 2024-06-13T01:06:51.502 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 20 MB 00:05 2024-06-13T01:06:55.731 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.3 MB 00:01 2024-06-13T01:06:56.360 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.3 MB 00:01 2024-06-13T01:06:57.965 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 17 kB 00:00 2024-06-13T01:06:58.193 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 17 kB 00:00 2024-06-13T01:06:59.623 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-06-13T01:06:59.867 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-06-13T01:07:04.910 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-06-13T01:07:05.215 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-13T01:07:06.958 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-13T01:07:06.959 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-13T01:07:07.036 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:07:07.045 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:07:07.046 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 81 M 2024-06-13T01:07:07.047 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 124 M 2024-06-13T01:07:07.047 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:07:07.257 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:07:07.259 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:07:07.264 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:07:07.271 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-13T01:07:07.271 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-13T01:07:07.366 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:07:07.374 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:07:07.375 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:07:07.375 INFO:teuthology.orchestra.run.smithi132.stdout:Install 4 Packages 2024-06-13T01:07:07.375 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:07:07.375 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 81 M 2024-06-13T01:07:07.375 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 124 M 2024-06-13T01:07:07.375 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:07:07.524 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] kernel-core-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:07:07.525 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] kernel-modules-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:07:07.531 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] kernel-5.14.0-457.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-457.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:07:07.561 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 2.5 MB/s | 989 kB 00:00 2024-06-13T01:07:07.841 INFO:teuthology.orchestra.run.smithi132.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 2.4 MB/s | 989 kB 00:00 2024-06-13T01:07:09.342 INFO:teuthology.orchestra.run.smithi132.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 8.4 MB/s | 16 MB 00:01 2024-06-13T01:07:10.803 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 4.4 MB/s | 16 MB 00:03 2024-06-13T01:07:11.184 INFO:teuthology.orchestra.run.smithi132.stdout:(3/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 9.3 MB/s | 35 MB 00:03 2024-06-13T01:07:11.442 INFO:teuthology.orchestra.run.smithi132.stdout:(4/4): kernel-modules-core-5.14.0-457.el9.x86_6 8.1 MB/s | 29 MB 00:03 2024-06-13T01:07:11.444 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:07:11.445 INFO:teuthology.orchestra.run.smithi132.stdout:Total 20 MB/s | 81 MB 00:04 2024-06-13T01:07:12.147 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:07:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:07:12.484 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:07:13.303 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 5.7 MB/s | 35 MB 00:06 2024-06-13T01:07:13.448 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:07:13.449 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:07:15.045 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-core-5.14.0-457.el9.x86_6 3.9 MB/s | 29 MB 00:07 2024-06-13T01:07:15.047 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:07:15.047 INFO:teuthology.orchestra.run.smithi155.stdout:Total 10 MB/s | 81 MB 00:08 2024-06-13T01:07:15.750 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:07:15.867 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:07:16.128 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:07:16.128 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:07:16.538 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-13T01:07:16.643 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T01:07:17.112 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:07:17.112 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:07:19.287 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T01:07:19.434 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T01:07:19.538 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:07:20.187 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-13T01:07:20.277 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T01:07:22.944 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T01:07:23.110 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T01:07:24.178 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T01:07:24.232 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:27.880 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T01:07:27.923 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:29.006 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:32.656 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:58.710 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:58.749 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:59.180 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:59.180 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-13T01:07:59.180 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T01:07:59.180 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-13T01:07:59.995 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:07:59.996 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:08:00.349 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T01:08:00.422 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T01:08:00.422 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T01:08:00.422 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-13T01:08:00.422 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-13T01:08:00.422 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-13T01:08:00.423 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-13T01:08:00.423 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2024-06-13T01:08:00.423 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T01:08:00.476 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T01:08:00.476 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T01:08:00.476 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-06-13T01:08:00.507 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-06-13T01:08:00.507 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-06-13T01:08:00.507 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2024-06-13T01:08:00.507 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2024-06-13T01:08:00.507 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-13T01:08:00.508 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:08:00.509 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-06-13T01:08:00.510 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-13T01:08:00.510 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 5.14.0-457.el9.x86_64 2024-06-13T01:08:00.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-13T01:08:01.096 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-06-13T01:08:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:08:02.036 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:08:02.432 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:08:02.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-13T01:08:02.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-13T01:08:02.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-13T01:08:02.539 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-13T01:08:02.559 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-06-13T01:08:02.561 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-06-13T01:08:02.577 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.xt42Wu7dat 2024-06-13T01:08:02.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xt42Wu7dat 2024-06-13T01:08:02.649 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.xt42Wu7dat 2024-06-13T01:08:02.786 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.xt42Wu7dat is 6KB 2024-06-13T01:08:02.834 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.xt42Wu7dat 2024-06-13T01:08:02.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-06-13T01:08:02.910 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-13T01:08:02.911 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-13T01:08:02.911 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-13T01:08:02.911 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-13T01:08:02.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-06-13T01:08:03.281 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:08:03.282 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:08:03.684 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T01:08:03.759 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.760 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T01:08:03.760 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-13T01:08:03.760 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-13T01:08:03.760 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-13T01:08:03.760 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-13T01:08:03.760 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-06-13T01:08:03.760 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-06-13T01:08:03.813 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-13T01:08:03.813 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-13T01:08:03.813 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-06-13T01:08:03.843 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-06-13T01:08:03.843 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-06-13T01:08:03.844 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:08:03.845 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-06-13T01:08:03.846 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-13T01:08:03.846 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-457.el9.x86_64 2024-06-13T01:08:03.846 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-13T01:08:04.469 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-06-13T01:08:05.893 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-13T01:08:05.924 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-06-13T01:08:05.926 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-06-13T01:08:05.941 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.FWHd8a9hBe 2024-06-13T01:08:05.941 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FWHd8a9hBe 2024-06-13T01:08:06.007 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.FWHd8a9hBe 2024-06-13T01:08:06.142 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.FWHd8a9hBe is 6KB 2024-06-13T01:08:06.162 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.FWHd8a9hBe 2024-06-13T01:08:06.177 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-06-13T01:08:06.239 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-13T01:08:06.239 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-13T01:08:06.240 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-13T01:08:06.240 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-13T01:08:06.241 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-13T01:08:06.406 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-06-13T01:08:33.086 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-13T01:08:33.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-06-13T01:08:33.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:08:36.412 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-13T01:08:36.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:08:36.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:08:51.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-06-13T01:08:54.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-06-13T01:09:00.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-06-13T01:09:00.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:09:03.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:09:03.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:09:07.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-06-13T01:09:19.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:09:19.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:10:00.657 DEBUG:teuthology.orchestra.remote:timed out 2024-06-13T01:10:12.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-06-13T01:10:12.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:10:13.055 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-06-13T01:10:13.413 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-06-13T01:10:13.413 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-457.el9.x86_64"... 2024-06-13T01:10:13.413 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-06-13T01:10:13.429 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-457.el9.x86_64 2024-06-13T01:10:13.430 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-13T01:10:13.430 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-13T01:10:13.430 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-13T01:10:14.431 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-13T01:10:14.431 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-13T01:10:14.506 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-06-13T01:10:14.540 DEBUG:teuthology.parallel:result is None 2024-06-13T01:10:19.020 DEBUG:teuthology.orchestra.remote:timed out 2024-06-13T01:10:34.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:10:34.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:10:34.299 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-06-13T01:10:34.681 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:10:34.682 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-457.el9.x86_64"... 2024-06-13T01:10:34.682 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-06-13T01:10:34.698 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-457.el9.x86_64 2024-06-13T01:10:34.698 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-13T01:10:34.698 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-13T01:10:34.698 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-13T01:10:35.699 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-13T01:10:35.699 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-13T01:10:36.068 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-06-13T01:10:36.100 DEBUG:teuthology.parallel:result is None 2024-06-13T01:10:36.101 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-13T01:10:36.108 INFO:teuthology.task.internal:Creating test directory... 2024-06-13T01:10:36.108 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-13T01:10:36.111 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-13T01:10:36.129 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-13T01:10:36.135 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-13T01:10:36.141 INFO:teuthology.task.internal:Creating archive directory... 2024-06-13T01:10:36.141 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-13T01:10:36.170 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-13T01:10:36.240 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-13T01:10:36.247 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-13T01:10:36.247 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-13T01:10:36.250 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-13T01:10:36.284 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T01:10:36.294 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T01:10:36.295 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T01:10:36.306 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-13T01:10:36.308 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-13T01:10:36.314 INFO:teuthology.task.internal:Configuring sudo... 2024-06-13T01:10:36.314 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-13T01:10:36.340 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-13T01:10:36.368 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-13T01:10:36.377 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-13T01:10:36.377 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-13T01:10:36.409 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-13T01:10:36.425 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T01:10:36.507 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T01:10:36.576 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T01:10:36.620 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T01:10:36.660 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:10:36.660 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-13T01:10:36.725 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T01:10:36.771 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-13T01:10:36.826 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T01:10:36.911 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-13T01:10:36.954 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:10:36.954 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-13T01:10:37.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-06-13T01:10:37.024 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-06-13T01:10:37.064 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T01:10:37.107 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T01:10:37.499 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-13T01:10:37.507 INFO:teuthology.task.internal:Starting timer... 2024-06-13T01:10:37.507 INFO:teuthology.run_tasks:Running task pcp... 2024-06-13T01:10:37.522 INFO:teuthology.run_tasks:Running task selinux... 2024-06-13T01:10:37.530 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-13T01:10:37.530 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-06-13T01:10:37.604 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-06-13T01:10:37.606 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-06-13T01:10:37.677 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-06-13T01:10:37.679 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-13T01:10:37.680 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-06-13T01:10:37.705 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-06-13T01:10:37.705 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-06-13T01:10:37.736 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-06-13T01:10:37.736 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-06-13T01:10:37.736 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T01:10:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:10:37.771 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T01:10:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:10:37.805 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-13T01:10:37.806 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-06-13T01:10:37.837 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-06-13T01:10:37.873 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-13T01:10:37.883 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-13T01:10:37.885 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-13T01:10:37.995 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-13T01:10:38.006 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-13T01:10:38.007 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi132.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-13T01:14:55.892 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-06-13T01:14:55.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-06-13T01:14:55.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:14:55.968 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-06-13T01:14:56.053 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-06-13T01:14:56.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:14:56.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-13T01:14:56.130 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-06-13T01:14:56.209 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-06-13T01:14:56.209 INFO:teuthology.run_tasks:Running task clock... 2024-06-13T01:14:56.219 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-13T01:14:56.220 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-13T01:14:56.220 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T01:14:56.222 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-13T01:14:56.222 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T01:14:56.249 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-13T01:14:56.263 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-13T01:14:56.277 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-13T01:14:56.286 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-06-13T01:14:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-13T01:14:56.297 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-06-13T01:14:56.311 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-13T01:14:56.315 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-06-13T01:14:56.325 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-13T01:14:56.326 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-06-13T01:14:56.340 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-13T01:14:56.355 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-13T01:14:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-06-13T01:14:56.383 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T01:14:56.383 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-06-13T01:14:56.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.383 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.410 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-06-13T01:14:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T01:14:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-06-13T01:14:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:14:56.415 INFO:teuthology.run_tasks:Running task install... 2024-06-13T01:14:56.425 DEBUG:teuthology.task.install:project ceph 2024-06-13T01:14:56.426 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-13T01:14:56.426 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-13T01:14:56.426 INFO:teuthology.task.install:Using flavor: default 2024-06-13T01:14:56.431 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-13T01:14:56.431 INFO:teuthology.task.install:extra packages: [] 2024-06-13T01:14:56.432 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-13T01:14:56.432 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:14:56.432 INFO:teuthology.packaging:ref: None 2024-06-13T01:14:56.432 INFO:teuthology.packaging:tag: None 2024-06-13T01:14:56.432 INFO:teuthology.packaging:branch: quincy 2024-06-13T01:14:56.432 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:14:56.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-13T01:14:56.434 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-13T01:14:56.435 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:14:56.435 INFO:teuthology.packaging:ref: None 2024-06-13T01:14:56.435 INFO:teuthology.packaging:tag: None 2024-06-13T01:14:56.435 INFO:teuthology.packaging:branch: quincy 2024-06-13T01:14:56.435 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:14:56.435 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-13T01:14:56.671 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/ 2024-06-13T01:14:56.671 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1006.g296b94f3 2024-06-13T01:14:56.736 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/ 2024-06-13T01:14:56.736 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1006.g296b94f3 2024-06-13T01:14:56.844 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-13T01:14:56.844 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:14:56.844 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-13T01:14:56.871 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-13T01:14:56.871 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:14:56.871 INFO:teuthology.packaging:ref: None 2024-06-13T01:14:56.871 INFO:teuthology.packaging:tag: None 2024-06-13T01:14:56.872 INFO:teuthology.packaging:branch: quincy 2024-06-13T01:14:56.872 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:14:56.872 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-13T01:14:56.910 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/296b94f31805b13f169a9221a78e5c5d2902e4d0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-13T01:14:56.911 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:14:56.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-13T01:14:56.937 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-13T01:14:56.937 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:14:56.937 INFO:teuthology.packaging:ref: None 2024-06-13T01:14:56.937 INFO:teuthology.packaging:tag: None 2024-06-13T01:14:56.937 INFO:teuthology.packaging:branch: quincy 2024-06-13T01:14:56.938 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:14:56.938 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-13T01:14:56.939 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-13T01:14:57.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-13T01:14:57.015 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-13T01:14:57.079 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-13T01:14:57.080 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-06-13T01:14:57.081 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-06-13T01:14:57.143 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-06-13T01:14:57.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-06-13T01:14:57.313 INFO:teuthology.orchestra.run.smithi155.stdout:45 files removed 2024-06-13T01:14:57.340 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-06-13T01:14:57.369 INFO:teuthology.orchestra.run.smithi132.stdout:45 files removed 2024-06-13T01:14:57.396 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2024-06-13T01:14:58.035 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 187 kB/s | 77 kB 00:00 2024-06-13T01:14:58.075 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 196 kB/s | 77 kB 00:00 2024-06-13T01:14:58.419 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-06-13T01:14:58.509 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-06-13T01:14:58.786 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-06-13T01:14:58.884 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 5.4 kB/s | 1.8 kB 00:00 2024-06-13T01:14:59.488 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-06-13T01:15:02.670 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.1 MB 00:03 2024-06-13T01:15:08.329 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 20 MB 00:04 2024-06-13T01:15:12.620 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.3 MB 00:00 2024-06-13T01:15:14.146 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-06-13T01:15:15.735 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-06-13T01:15:16.788 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-06-13T01:15:21.039 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-06-13T01:15:21.496 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.3 MB 00:01 2024-06-13T01:15:23.113 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 17 kB 00:00 2024-06-13T01:15:23.430 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:15:23.446 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:15:23.446 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:15:23.446 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:15:23.446 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:15:23.446 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 14 M 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 3.1 M 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 2.9 M 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 5.0 M 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 22 M 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 25 k 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 638 k 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-13T01:15:23.447 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 458 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 4.9 M 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 45 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 118 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 156 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 320 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 292 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 100 k 2024-06-13T01:15:23.448 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout:Install 22 Packages 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:23.449 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 58 M 2024-06-13T01:15:23.450 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:15:24.832 INFO:teuthology.orchestra.run.smithi132.stdout:(1/24): ceph-base-17.2.7-1006.g296b94f3.el9.x86 6.1 MB/s | 5.0 MB 00:00 2024-06-13T01:15:24.890 INFO:teuthology.orchestra.run.smithi132.stdout:(2/24): ceph-selinux-17.2.7-1006.g296b94f3.el9. 432 kB/s | 25 kB 00:00 2024-06-13T01:15:25.007 INFO:teuthology.orchestra.run.smithi132.stdout:(3/24): libcephfs2-17.2.7-1006.g296b94f3.el9.x8 5.3 MB/s | 638 kB 00:00 2024-06-13T01:15:25.091 INFO:teuthology.orchestra.run.smithi132.stdout:(4/24): libradosstriper1-17.2.7-1006.g296b94f3. 5.4 MB/s | 458 kB 00:00 2024-06-13T01:15:25.157 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-06-13T01:15:25.408 INFO:teuthology.orchestra.run.smithi132.stdout:(5/24): ceph-radosgw-17.2.7-1006.g296b94f3.el9. 10 MB/s | 14 MB 00:01 2024-06-13T01:15:25.492 INFO:teuthology.orchestra.run.smithi132.stdout:(6/24): librgw2-17.2.7-1006.g296b94f3.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-06-13T01:15:25.517 INFO:teuthology.orchestra.run.smithi132.stdout:(7/24): python3-ceph-argparse-17.2.7-1006.g296b 416 kB/s | 45 kB 00:00 2024-06-13T01:15:25.550 INFO:teuthology.orchestra.run.smithi132.stdout:(8/24): python3-ceph-common-17.2.7-1006.g296b94 2.0 MB/s | 118 kB 00:00 2024-06-13T01:15:25.567 INFO:teuthology.orchestra.run.smithi132.stdout:(9/24): python3-cephfs-17.2.7-1006.g296b94f3.el 3.0 MB/s | 156 kB 00:00 2024-06-13T01:15:25.601 INFO:teuthology.orchestra.run.smithi132.stdout:(10/24): python3-rados-17.2.7-1006.g296b94f3.el 6.2 MB/s | 320 kB 00:00 2024-06-13T01:15:25.626 INFO:teuthology.orchestra.run.smithi132.stdout:(11/24): python3-rbd-17.2.7-1006.g296b94f3.el9. 4.9 MB/s | 292 kB 00:00 2024-06-13T01:15:25.652 INFO:teuthology.orchestra.run.smithi132.stdout:(12/24): python3-rgw-17.2.7-1006.g296b94f3.el9. 1.9 MB/s | 100 kB 00:00 2024-06-13T01:15:25.707 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:25.715 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:25.827 INFO:teuthology.orchestra.run.smithi132.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-06-13T01:15:25.829 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:25.877 INFO:teuthology.orchestra.run.smithi132.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 639 kB/s | 160 kB 00:00 2024-06-13T01:15:25.995 INFO:teuthology.orchestra.run.smithi132.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-06-13T01:15:26.036 INFO:teuthology.orchestra.run.smithi132.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-06-13T01:15:26.087 INFO:teuthology.orchestra.run.smithi132.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.0 MB/s | 309 kB 00:00 2024-06-13T01:15:26.112 INFO:teuthology.orchestra.run.smithi132.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-13T01:15:26.138 INFO:teuthology.orchestra.run.smithi132.stdout:(19/24): python3-prettytable-0.7.2-27.el9.noarc 296 kB/s | 42 kB 00:00 2024-06-13T01:15:26.163 INFO:teuthology.orchestra.run.smithi132.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-13T01:15:26.305 INFO:teuthology.orchestra.run.smithi132.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-06-13T01:15:26.372 INFO:teuthology.orchestra.run.smithi132.stdout:(22/24): librados2-17.2.7-1006.g296b94f3.el9.x8 15 MB/s | 3.1 MB 00:00 2024-06-13T01:15:26.539 INFO:teuthology.orchestra.run.smithi132.stdout:(23/24): librbd1-17.2.7-1006.g296b94f3.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-06-13T01:15:30.005 INFO:teuthology.orchestra.run.smithi132.stdout:(24/24): ceph-common-17.2.7-1006.g296b94f3.el9. 3.8 MB/s | 22 MB 00:05 2024-06-13T01:15:30.009 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:15:30.010 INFO:teuthology.orchestra.run.smithi132.stdout:Total 8.8 MB/s | 58 MB 00:06 2024-06-13T01:15:30.069 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:15:30.101 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:15:30.101 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:15:30.455 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-06-13T01:15:30.627 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:15:30.627 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:15:31.814 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:15:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-13T01:15:32.241 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 2/26 2024-06-13T01:15:32.341 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 2/26 2024-06-13T01:15:32.428 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 3/26 2024-06-13T01:15:32.451 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 4/26 2024-06-13T01:15:32.543 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 4/26 2024-06-13T01:15:32.586 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-13T01:15:32.615 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-13T01:15:32.879 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:15:32.892 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 14 M 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 3.1 M 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 2.9 M 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 5.0 M 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 22 M 2024-06-13T01:15:32.893 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 25 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 638 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 458 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 4.9 M 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 45 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 118 k 2024-06-13T01:15:32.894 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 156 k 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 320 k 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 292 k 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 100 k 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout:Install 22 Packages 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-06-13T01:15:32.895 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:32.896 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 58 M 2024-06-13T01:15:32.896 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:15:33.048 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-13T01:15:33.079 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 8/26 2024-06-13T01:15:33.142 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 8/26 2024-06-13T01:15:33.170 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-13T01:15:33.213 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:17.2.7-1006.g296b94f3.el9. 10/26 2024-06-13T01:15:33.268 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 11/26 2024-06-13T01:15:33.338 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 12/26 2024-06-13T01:15:33.365 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_6 13/26 2024-06-13T01:15:33.431 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_6 13/26 2024-06-13T01:15:33.472 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-13T01:15:33.541 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-13T01:15:33.573 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-13T01:15:33.869 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-13T01:15:33.904 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 18/26 2024-06-13T01:15:33.976 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 18/26 2024-06-13T01:15:34.051 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 19/26 2024-06-13T01:15:34.081 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:17.2.7-1006.g296b94f3.el9.x8 20/26 2024-06-13T01:15:34.140 INFO:teuthology.orchestra.run.smithi155.stdout:(1/24): ceph-radosgw-17.2.7-1006.g296b94f3.el9. 14 MB/s | 14 MB 00:01 2024-06-13T01:15:34.190 INFO:teuthology.orchestra.run.smithi155.stdout:(2/24): ceph-selinux-17.2.7-1006.g296b94f3.el9. 506 kB/s | 25 kB 00:00 2024-06-13T01:15:34.273 INFO:teuthology.orchestra.run.smithi155.stdout:(3/24): libcephfs2-17.2.7-1006.g296b94f3.el9.x8 7.5 MB/s | 638 kB 00:00 2024-06-13T01:15:34.332 INFO:teuthology.orchestra.run.smithi155.stdout:(4/24): libradosstriper1-17.2.7-1006.g296b94f3. 7.7 MB/s | 458 kB 00:00 2024-06-13T01:15:34.466 INFO:teuthology.orchestra.run.smithi155.stdout:(5/24): ceph-base-17.2.7-1006.g296b94f3.el9.x86 3.7 MB/s | 5.0 MB 00:01 2024-06-13T01:15:34.566 INFO:teuthology.orchestra.run.smithi155.stdout:(6/24): python3-ceph-argparse-17.2.7-1006.g296b 452 kB/s | 45 kB 00:00 2024-06-13T01:15:34.633 INFO:teuthology.orchestra.run.smithi155.stdout:(7/24): librgw2-17.2.7-1006.g296b94f3.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-06-13T01:15:34.658 INFO:teuthology.orchestra.run.smithi155.stdout:(8/24): python3-ceph-common-17.2.7-1006.g296b94 1.2 MB/s | 118 kB 00:00 2024-06-13T01:15:34.683 INFO:teuthology.orchestra.run.smithi155.stdout:(9/24): python3-cephfs-17.2.7-1006.g296b94f3.el 3.1 MB/s | 156 kB 00:00 2024-06-13T01:15:34.726 INFO:teuthology.orchestra.run.smithi155.stdout:(10/24): python3-rados-17.2.7-1006.g296b94f3.el 4.7 MB/s | 320 kB 00:00 2024-06-13T01:15:34.750 INFO:teuthology.orchestra.run.smithi155.stdout:(11/24): python3-rbd-17.2.7-1006.g296b94f3.el9. 4.3 MB/s | 292 kB 00:00 2024-06-13T01:15:34.801 INFO:teuthology.orchestra.run.smithi155.stdout:(12/24): python3-rgw-17.2.7-1006.g296b94f3.el9. 1.3 MB/s | 100 kB 00:00 2024-06-13T01:15:34.831 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:34.837 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:34.984 INFO:teuthology.orchestra.run.smithi155.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 247 kB/s | 45 kB 00:00 2024-06-13T01:15:35.010 INFO:teuthology.orchestra.run.smithi155.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 619 kB/s | 160 kB 00:00 2024-06-13T01:15:35.010 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:35.011 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:35.135 INFO:teuthology.orchestra.run.smithi155.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-06-13T01:15:35.161 INFO:teuthology.orchestra.run.smithi155.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-06-13T01:15:35.202 INFO:teuthology.orchestra.run.smithi155.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 632 kB/s | 42 kB 00:00 2024-06-13T01:15:35.236 INFO:teuthology.orchestra.run.smithi155.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-06-13T01:15:35.278 INFO:teuthology.orchestra.run.smithi155.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-06-13T01:15:35.303 INFO:teuthology.orchestra.run.smithi155.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-06-13T01:15:35.503 INFO:teuthology.orchestra.run.smithi155.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-06-13T01:15:35.895 INFO:teuthology.orchestra.run.smithi155.stdout:(22/24): librbd1-17.2.7-1006.g296b94f3.el9.x86_ 7.5 MB/s | 2.9 MB 00:00 2024-06-13T01:15:35.954 INFO:teuthology.orchestra.run.smithi155.stdout:(23/24): librados2-17.2.7-1006.g296b94f3.el9.x8 4.7 MB/s | 3.1 MB 00:00 2024-06-13T01:15:36.163 INFO:teuthology.orchestra.run.smithi155.stdout:(24/24): ceph-common-17.2.7-1006.g296b94f3.el9. 7.4 MB/s | 22 MB 00:03 2024-06-13T01:15:36.167 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:15:36.168 INFO:teuthology.orchestra.run.smithi155.stdout:Total 18 MB/s | 58 MB 00:03 2024-06-13T01:15:36.224 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:15:36.256 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:15:36.256 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:15:36.448 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 21/26 2024-06-13T01:15:36.486 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 21/26 2024-06-13T01:15:36.784 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:15:36.785 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:15:37.034 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 21/26 2024-06-13T01:15:37.074 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 22/26 2024-06-13T01:15:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 22/26 2024-06-13T01:15:37.166 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-13T01:15:37.166 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-13T01:15:37.166 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:37.192 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 23/26 2024-06-13T01:15:37.948 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:15:38.334 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-13T01:15:38.390 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 2/26 2024-06-13T01:15:38.482 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 2/26 2024-06-13T01:15:38.578 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 3/26 2024-06-13T01:15:38.600 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 4/26 2024-06-13T01:15:38.707 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 4/26 2024-06-13T01:15:38.776 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-13T01:15:38.805 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-13T01:15:39.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-13T01:15:39.294 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 8/26 2024-06-13T01:15:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 8/26 2024-06-13T01:15:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-13T01:15:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:17.2.7-1006.g296b94f3.el9. 10/26 2024-06-13T01:15:39.457 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 11/26 2024-06-13T01:15:39.519 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 12/26 2024-06-13T01:15:39.547 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_6 13/26 2024-06-13T01:15:39.622 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_6 13/26 2024-06-13T01:15:39.661 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-13T01:15:39.705 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-13T01:15:39.738 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-13T01:15:40.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-13T01:15:40.076 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 18/26 2024-06-13T01:15:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 18/26 2024-06-13T01:15:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 19/26 2024-06-13T01:15:40.254 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:17.2.7-1006.g296b94f3.el9.x8 20/26 2024-06-13T01:15:42.639 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 21/26 2024-06-13T01:15:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 21/26 2024-06-13T01:15:43.232 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 21/26 2024-06-13T01:15:43.270 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 22/26 2024-06-13T01:15:43.350 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 22/26 2024-06-13T01:15:43.350 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-13T01:15:43.350 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-13T01:15:43.350 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:43.379 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 23/26 2024-06-13T01:15:46.642 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 23/26 2024-06-13T01:15:46.642 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-06-13T01:15:46.642 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-06-13T01:15:46.642 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-06-13T01:15:46.643 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-06-13T01:15:46.643 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-06-13T01:15:46.643 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-06-13T01:15:46.643 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-06-13T01:15:46.643 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:47.955 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 24/26 2024-06-13T01:15:48.016 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 24/26 2024-06-13T01:15:48.017 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:15:48.017 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-13T01:15:48.017 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T01:15:48.017 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T01:15:48.017 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:48.018 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-13T01:15:48.107 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-13T01:15:48.108 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-13T01:15:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-13T01:15:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 1/26 2024-06-13T01:15:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 2/26 2024-06-13T01:15:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 3/26 2024-06-13T01:15:49.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 4/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 5/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_6 6/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 7/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1006.g296b94f3.el9. 8/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:17.2.7-1006.g296b94f3.el9.x8 9/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 10/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 11/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 12/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 13/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-13T01:15:49.356 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 23/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-13T01:15:49.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 25/26 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.072 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.073 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:50.074 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:15:50.426 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2024-06-13T01:15:51.003 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:30 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:15:51.127 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:15:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:15:51.128 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 48 M 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 163 k 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:Install 4 Packages 2024-06-13T01:15:51.129 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:15:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 48 M 2024-06-13T01:15:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 192 M 2024-06-13T01:15:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:15:51.479 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:51.697 INFO:teuthology.orchestra.run.smithi132.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.0 MB/s | 304 kB 00:00 2024-06-13T01:15:51.698 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:51.756 INFO:teuthology.orchestra.run.smithi132.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-13T01:15:51.814 INFO:teuthology.orchestra.run.smithi132.stdout:(3/4): libcephsqlite-17.2.7-1006.g296b94f3.el9. 396 kB/s | 163 kB 00:00 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 23/26 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-06-13T01:15:52.884 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-06-13T01:15:52.885 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:53.739 INFO:teuthology.orchestra.run.smithi132.stdout:(4/4): ceph-test-17.2.7-1006.g296b94f3.el9.x86_ 20 MB/s | 48 MB 00:02 2024-06-13T01:15:53.742 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:15:53.743 INFO:teuthology.orchestra.run.smithi132.stdout:Total 19 MB/s | 48 MB 00:02 2024-06-13T01:15:53.770 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:15:53.789 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:15:53.789 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:15:54.152 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:15:54.152 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:15:54.155 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 24/26 2024-06-13T01:15:54.234 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 24/26 2024-06-13T01:15:54.234 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:15:54.234 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-13T01:15:54.234 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T01:15:54.235 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-13T01:15:54.235 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:54.236 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-13T01:15:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-13T01:15:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-13T01:15:54.678 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:15:54.749 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-13T01:15:54.789 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-13T01:15:54.816 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 3/4 2024-06-13T01:15:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-13T01:15:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 1/26 2024-06-13T01:15:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 2/26 2024-06-13T01:15:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 3/26 2024-06-13T01:15:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 4/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 5/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_6 6/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 7/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1006.g296b94f3.el9. 8/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:17.2.7-1006.g296b94f3.el9.x8 9/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 10/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 11/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 12/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 13/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-13T01:15:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 23/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-13T01:15:55.555 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 25/26 2024-06-13T01:15:56.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-13T01:15:56.335 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-13T01:15:56.336 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:15:56.337 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-13T01:15:56.338 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:56.338 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:15:56.761 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-06-13T01:15:57.342 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:27 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:15:57.468 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:15:57.469 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:15:57.469 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:15:57.469 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:15:57.469 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:15:57.469 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 48 M 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 163 k 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-06-13T01:15:57.470 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:15:57.471 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 48 M 2024-06-13T01:15:57.471 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 192 M 2024-06-13T01:15:57.471 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:15:57.594 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:57.745 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): libcephsqlite-17.2.7-1006.g296b94f3.el9. 711 kB/s | 163 kB 00:00 2024-06-13T01:15:57.747 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:15:57.895 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 428 kB/s | 64 kB 00:00 2024-06-13T01:15:57.929 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 737 kB/s | 304 kB 00:00 2024-06-13T01:15:59.236 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 3/4 2024-06-13T01:15:59.307 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 4/4 2024-06-13T01:16:00.346 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): ceph-test-17.2.7-1006.g296b94f3.el9.x86_ 17 MB/s | 48 MB 00:02 2024-06-13T01:16:00.350 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:00.350 INFO:teuthology.orchestra.run.smithi155.stdout:Total 17 MB/s | 48 MB 00:02 2024-06-13T01:16:00.378 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:00.397 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:00.397 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:01.311 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:01.397 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-13T01:16:01.437 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-13T01:16:01.460 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 4/4 2024-06-13T01:16:01.460 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 1/4 2024-06-13T01:16:01.460 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 2/4 2024-06-13T01:16:01.460 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-13T01:16:01.473 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 3/4 2024-06-13T01:16:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-13T01:16:01.749 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:01.750 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:02.048 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2024-06-13T01:16:02.634 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:41 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:02.769 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:02.771 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 6.5 k 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 2.1 M 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 1.4 M 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 240 k 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 3.9 M 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 15 M 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-13T01:16:02.772 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-13T01:16:02.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-13T01:16:02.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout:Install 33 Packages 2024-06-13T01:16:02.775 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:02.776 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 27 M 2024-06-13T01:16:02.776 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 96 M 2024-06-13T01:16:02.776 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:06.017 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 3/4 2024-06-13T01:16:06.087 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 4/4 2024-06-13T01:16:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 4/4 2024-06-13T01:16:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 1/4 2024-06-13T01:16:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 2/4 2024-06-13T01:16:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-13T01:16:08.662 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-13T01:16:08.662 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:08.663 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:08.970 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-06-13T01:16:09.561 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:39 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:09.696 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:09.698 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 6.5 k 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 2.1 M 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 1.4 M 2024-06-13T01:16:09.699 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 240 k 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 3.9 M 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 15 M 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-13T01:16:09.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-13T01:16:09.701 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-13T01:16:09.702 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-13T01:16:09.702 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-13T01:16:09.702 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-13T01:16:09.702 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-13T01:16:09.702 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-13T01:16:09.703 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-13T01:16:09.704 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-13T01:16:09.704 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-13T01:16:09.704 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-13T01:16:09.704 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-13T01:16:09.704 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:09.705 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:09.705 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:09.705 INFO:teuthology.orchestra.run.smithi155.stdout:Install 33 Packages 2024-06-13T01:16:09.705 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:09.706 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 27 M 2024-06-13T01:16:09.706 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 96 M 2024-06-13T01:16:09.706 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:10.253 INFO:teuthology.orchestra.run.smithi155.stdout:(1/33): ceph-17.2.7-1006.g296b94f3.el9.x86_64.r 31 kB/s | 6.5 kB 00:00 2024-06-13T01:16:10.395 INFO:teuthology.orchestra.run.smithi155.stdout:(2/33): ceph-mgr-17.2.7-1006.g296b94f3.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-06-13T01:16:10.671 INFO:teuthology.orchestra.run.smithi155.stdout:(3/33): ceph-mds-17.2.7-1006.g296b94f3.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2024-06-13T01:16:10.788 INFO:teuthology.orchestra.run.smithi155.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1006.g296b 2.0 MB/s | 240 kB 00:00 2024-06-13T01:16:10.826 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:10.955 INFO:teuthology.orchestra.run.smithi155.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-06-13T01:16:10.956 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:11.063 INFO:teuthology.orchestra.run.smithi155.stdout:(6/33): ceph-mon-17.2.7-1006.g296b94f3.el9.x86_ 4.8 MB/s | 3.9 MB 00:00 2024-06-13T01:16:11.065 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:11.172 INFO:teuthology.orchestra.run.smithi155.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 985 kB/s | 106 kB 00:00 2024-06-13T01:16:11.339 INFO:teuthology.orchestra.run.smithi155.stdout:(8/33): ceph-osd-17.2.7-1006.g296b94f3.el9.x86_ 16 MB/s | 15 MB 00:00 2024-06-13T01:16:11.381 INFO:teuthology.orchestra.run.smithi155.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 2.9 MB/s | 1.2 MB 00:00 2024-06-13T01:16:11.406 INFO:teuthology.orchestra.run.smithi155.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 578 kB/s | 135 kB 00:00 2024-06-13T01:16:11.408 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:11.473 INFO:teuthology.orchestra.run.smithi155.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 935 kB/s | 125 kB 00:00 2024-06-13T01:16:11.498 INFO:teuthology.orchestra.run.smithi155.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-06-13T01:16:11.499 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:11.499 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:11.501 INFO:teuthology.orchestra.run.smithi132.stdout:(1/33): ceph-17.2.7-1006.g296b94f3.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-06-13T01:16:11.573 INFO:teuthology.orchestra.run.smithi155.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-06-13T01:16:11.599 INFO:teuthology.orchestra.run.smithi155.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-06-13T01:16:11.624 INFO:teuthology.orchestra.run.smithi155.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 406 kB/s | 51 kB 00:00 2024-06-13T01:16:11.649 INFO:teuthology.orchestra.run.smithi155.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-06-13T01:16:11.660 INFO:teuthology.orchestra.run.smithi132.stdout:(2/33): ceph-mds-17.2.7-1006.g296b94f3.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-06-13T01:16:11.700 INFO:teuthology.orchestra.run.smithi155.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-06-13T01:16:11.750 INFO:teuthology.orchestra.run.smithi155.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 454 kB/s | 11 kB 00:00 2024-06-13T01:16:11.775 INFO:teuthology.orchestra.run.smithi155.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-06-13T01:16:11.800 INFO:teuthology.orchestra.run.smithi155.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 367 kB/s | 18 kB 00:00 2024-06-13T01:16:11.825 INFO:teuthology.orchestra.run.smithi155.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-06-13T01:16:11.851 INFO:teuthology.orchestra.run.smithi155.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-06-13T01:16:11.876 INFO:teuthology.orchestra.run.smithi155.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-06-13T01:16:11.877 INFO:teuthology.orchestra.run.smithi132.stdout:(3/33): ceph-mgr-17.2.7-1006.g296b94f3.el9.x86_ 2.3 MB/s | 1.4 MB 00:00 2024-06-13T01:16:11.901 INFO:teuthology.orchestra.run.smithi155.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-06-13T01:16:11.926 INFO:teuthology.orchestra.run.smithi155.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-13T01:16:11.952 INFO:teuthology.orchestra.run.smithi155.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-06-13T01:16:11.977 INFO:teuthology.orchestra.run.smithi155.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-13T01:16:11.986 INFO:teuthology.orchestra.run.smithi132.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1006.g296b 2.2 MB/s | 240 kB 00:00 2024-06-13T01:16:12.002 INFO:teuthology.orchestra.run.smithi155.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-13T01:16:12.019 INFO:teuthology.orchestra.run.smithi155.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 861 kB/s | 36 kB 00:00 2024-06-13T01:16:12.027 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:12.044 INFO:teuthology.orchestra.run.smithi155.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-06-13T01:16:12.070 INFO:teuthology.orchestra.run.smithi155.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-06-13T01:16:12.087 INFO:teuthology.orchestra.run.smithi155.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-06-13T01:16:12.112 INFO:teuthology.orchestra.run.smithi155.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-06-13T01:16:12.116 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:12.116 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 27 MB 00:02 2024-06-13T01:16:12.226 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:12.236 INFO:teuthology.orchestra.run.smithi132.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-06-13T01:16:12.237 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:12.262 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:12.262 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:12.369 INFO:teuthology.orchestra.run.smithi132.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 9.2 MB/s | 1.2 MB 00:00 2024-06-13T01:16:12.370 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:12.428 INFO:teuthology.orchestra.run.smithi132.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-06-13T01:16:12.487 INFO:teuthology.orchestra.run.smithi132.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-13T01:16:12.632 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:12.632 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:12.670 INFO:teuthology.orchestra.run.smithi132.stdout:(9/33): ceph-osd-17.2.7-1006.g296b94f3.el9.x86_ 15 MB/s | 15 MB 00:01 2024-06-13T01:16:12.696 INFO:teuthology.orchestra.run.smithi132.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-06-13T01:16:12.697 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:12.871 INFO:teuthology.orchestra.run.smithi132.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-06-13T01:16:12.874 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:12.913 INFO:teuthology.orchestra.run.smithi132.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 794 kB/s | 172 kB 00:00 2024-06-13T01:16:12.916 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:12.972 INFO:teuthology.orchestra.run.smithi132.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 874 kB/s | 51 kB 00:00 2024-06-13T01:16:12.997 INFO:teuthology.orchestra.run.smithi132.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-06-13T01:16:13.022 INFO:teuthology.orchestra.run.smithi132.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-06-13T01:16:13.047 INFO:teuthology.orchestra.run.smithi132.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-06-13T01:16:13.073 INFO:teuthology.orchestra.run.smithi132.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-06-13T01:16:13.123 INFO:teuthology.orchestra.run.smithi132.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 455 kB/s | 11 kB 00:00 2024-06-13T01:16:13.156 INFO:teuthology.orchestra.run.smithi132.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 574 kB/s | 18 kB 00:00 2024-06-13T01:16:13.182 INFO:teuthology.orchestra.run.smithi132.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 953 kB/s | 23 kB 00:00 2024-06-13T01:16:13.207 INFO:teuthology.orchestra.run.smithi132.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 785 kB/s | 19 kB 00:00 2024-06-13T01:16:13.232 INFO:teuthology.orchestra.run.smithi132.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 786 kB/s | 20 kB 00:00 2024-06-13T01:16:13.258 INFO:teuthology.orchestra.run.smithi132.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-06-13T01:16:13.283 INFO:teuthology.orchestra.run.smithi132.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-06-13T01:16:13.308 INFO:teuthology.orchestra.run.smithi132.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-13T01:16:13.334 INFO:teuthology.orchestra.run.smithi132.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 653 kB/s | 16 kB 00:00 2024-06-13T01:16:13.359 INFO:teuthology.orchestra.run.smithi132.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-06-13T01:16:13.384 INFO:teuthology.orchestra.run.smithi132.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-13T01:16:13.410 INFO:teuthology.orchestra.run.smithi132.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 706 kB/s | 36 kB 00:00 2024-06-13T01:16:13.443 INFO:teuthology.orchestra.run.smithi132.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-06-13T01:16:13.468 INFO:teuthology.orchestra.run.smithi132.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 811 kB/s | 20 kB 00:00 2024-06-13T01:16:13.494 INFO:teuthology.orchestra.run.smithi132.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-06-13T01:16:13.752 INFO:teuthology.orchestra.run.smithi132.stdout:(33/33): ceph-mon-17.2.7-1006.g296b94f3.el9.x86 1.7 MB/s | 3.9 MB 00:02 2024-06-13T01:16:13.756 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:13.756 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.5 MB/s | 27 MB 00:10 2024-06-13T01:16:13.781 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:13.839 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-13T01:16:13.867 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:13.871 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-13T01:16:13.904 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:13.904 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:13.913 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-13T01:16:14.006 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-13T01:16:14.108 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-13T01:16:14.149 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-13T01:16:14.184 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-13T01:16:14.214 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-13T01:16:14.248 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-13T01:16:14.281 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-13T01:16:14.282 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:14.282 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-13T01:16:14.349 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-13T01:16:14.383 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-13T01:16:14.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-13T01:16:14.494 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-13T01:16:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-13T01:16:14.712 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-13T01:16:14.765 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-13T01:16:14.804 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-13T01:16:14.853 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-13T01:16:14.918 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-13T01:16:15.106 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-13T01:16:15.157 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-13T01:16:15.236 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-13T01:16:15.326 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-13T01:16:15.376 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-13T01:16:15.436 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:15.499 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-13T01:16:15.505 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-13T01:16:15.537 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-13T01:16:15.570 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-13T01:16:15.648 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1006.g296b94f3.el9. 28/33 2024-06-13T01:16:15.661 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-13T01:16:15.677 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 29/33 2024-06-13T01:16:15.703 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 29/33 2024-06-13T01:16:15.703 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:15.703 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-13T01:16:15.703 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T01:16:15.704 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T01:16:15.704 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:15.757 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-13T01:16:15.806 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-13T01:16:15.842 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-13T01:16:15.871 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-13T01:16:15.906 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-13T01:16:15.956 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-13T01:16:15.997 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-13T01:16:16.031 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-13T01:16:16.066 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-13T01:16:16.104 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-13T01:16:16.177 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-13T01:16:16.323 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-13T01:16:16.404 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-13T01:16:16.457 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-13T01:16:16.496 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-13T01:16:16.545 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-13T01:16:16.617 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-13T01:16:16.820 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-13T01:16:16.867 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-13T01:16:16.936 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-13T01:16:17.028 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 30/33 2024-06-13T01:16:17.034 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-13T01:16:17.056 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 30/33 2024-06-13T01:16:17.056 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:17.056 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-13T01:16:17.056 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T01:16:17.056 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T01:16:17.056 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:17.074 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-13T01:16:17.191 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-13T01:16:17.345 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1006.g296b94f3.el9. 28/33 2024-06-13T01:16:17.377 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 29/33 2024-06-13T01:16:17.402 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 29/33 2024-06-13T01:16:17.403 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:17.403 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-13T01:16:17.403 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T01:16:17.403 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-13T01:16:17.403 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:17.420 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 31/33 2024-06-13T01:16:17.444 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 31/33 2024-06-13T01:16:17.444 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:17.444 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-13T01:16:17.444 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T01:16:17.444 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T01:16:17.444 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:17.654 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 32/33 2024-06-13T01:16:17.678 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 32/33 2024-06-13T01:16:17.678 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:17.679 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-13T01:16:17.679 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T01:16:17.679 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T01:16:17.679 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:17.743 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 33/33 2024-06-13T01:16:18.719 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 30/33 2024-06-13T01:16:18.746 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 30/33 2024-06-13T01:16:18.746 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:18.747 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-13T01:16:18.747 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T01:16:18.747 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-13T01:16:18.747 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:19.112 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 31/33 2024-06-13T01:16:19.136 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 31/33 2024-06-13T01:16:19.136 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:19.136 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-13T01:16:19.136 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T01:16:19.136 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-13T01:16:19.136 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 33/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 1/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 2/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 3/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 4/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 5/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1006.g296b94f3.el9. 6/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-13T01:16:19.292 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-13T01:16:19.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-13T01:16:19.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-13T01:16:19.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-13T01:16:19.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-13T01:16:19.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-13T01:16:19.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-13T01:16:19.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-13T01:16:19.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-13T01:16:19.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-13T01:16:19.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-13T01:16:19.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-13T01:16:19.346 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 32/33 2024-06-13T01:16:19.369 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 32/33 2024-06-13T01:16:19.370 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:19.370 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-13T01:16:19.370 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T01:16:19.370 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-13T01:16:19.370 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:19.443 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 33/33 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-13T01:16:20.330 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-13T01:16:20.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-13T01:16:20.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-13T01:16:20.333 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-13T01:16:20.333 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-13T01:16:20.333 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:20.333 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:20.666 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 33/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 1/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 2/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 3/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 4/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 5/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1006.g296b94f3.el9. 6/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-13T01:16:21.102 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-13T01:16:21.103 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-13T01:16:21.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-13T01:16:21.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-13T01:16:21.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-13T01:16:21.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-13T01:16:21.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-13T01:16:21.104 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-13T01:16:21.105 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-13T01:16:21.105 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-13T01:16:21.105 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-13T01:16:21.105 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-13T01:16:21.105 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-13T01:16:21.247 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:51 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:21.330 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:16:21.372 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:21.373 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:16:21.373 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:21.438 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-06-13T01:16:22.009 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:52 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:22.113 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-13T01:16:22.114 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-13T01:16:22.115 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-13T01:16:22.116 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-13T01:16:22.116 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-13T01:16:22.116 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-13T01:16:22.116 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:22.116 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:22.135 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 82 k 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:16:22.136 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:22.137 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 82 k 2024-06-13T01:16:22.137 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 372 k 2024-06-13T01:16:22.137 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:22.482 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2024-06-13T01:16:22.752 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-17.2.7-1006.g296b94f3.el9.noarch.rpm 133 kB/s | 82 kB 00:00 2024-06-13T01:16:22.753 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:22.753 INFO:teuthology.orchestra.run.smithi155.stdout:Total 133 kB/s | 82 kB 00:00 2024-06-13T01:16:22.753 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:22.762 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:22.763 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:22.763 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:22.763 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:22.831 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:23.065 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:02 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:23.150 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:16:23.192 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:23.193 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:16:23.193 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:23.253 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:23.260 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2024-06-13T01:16:23.356 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:23.733 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:23.837 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:02 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:23.962 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:23.962 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 82 k 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 82 k 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 372 k 2024-06-13T01:16:23.963 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:24.002 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:24.002 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:24.003 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:24.003 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:24.003 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:24.003 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:24.225 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-06-13T01:16:24.437 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-17.2.7-1006.g296b94f3.el9.noarch.rpm 173 kB/s | 82 kB 00:00 2024-06-13T01:16:24.438 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:24.438 INFO:teuthology.orchestra.run.smithi132.stdout:Total 173 kB/s | 82 kB 00:00 2024-06-13T01:16:24.438 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:24.443 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:24.443 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:24.445 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:24.445 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:24.516 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:24.813 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:54 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 138 k 2024-06-13T01:16:24.946 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:24.947 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:24.947 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:24.947 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:16:24.947 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:24.947 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 138 k 2024-06-13T01:16:24.947 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 379 k 2024-06-13T01:16:24.948 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:24.986 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:25.076 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:25.258 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-17.2.7-1006.g296b94 443 kB/s | 138 kB 00:00 2024-06-13T01:16:25.259 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:25.260 INFO:teuthology.orchestra.run.smithi155.stdout:Total 441 kB/s | 138 kB 00:00 2024-06-13T01:16:25.260 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:25.266 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:25.266 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:25.322 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:25.322 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:25.443 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:25.459 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:25.509 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3. 1/1 2024-06-13T01:16:25.534 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3. 1/1 2024-06-13T01:16:25.534 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:25.534 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-13T01:16:25.534 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:25.720 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/1 2024-06-13T01:16:25.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:25.720 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:25.720 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:25.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:25.720 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:25.940 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2024-06-13T01:16:26.503 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3. 1/1 2024-06-13T01:16:26.503 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:26.503 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:26.503 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:26.503 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:26.503 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:26.522 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:05 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:26.654 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 138 k 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:26.655 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 138 k 2024-06-13T01:16:26.656 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 379 k 2024-06-13T01:16:26.656 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:26.694 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-06-13T01:16:26.885 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-17.2.7-1006.g296b94 600 kB/s | 138 kB 00:00 2024-06-13T01:16:26.886 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:26.886 INFO:teuthology.orchestra.run.smithi132.stdout:Total 595 kB/s | 138 kB 00:00 2024-06-13T01:16:26.886 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:26.894 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:26.894 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:26.949 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:26.949 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:27.093 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:27.145 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3. 1/1 2024-06-13T01:16:27.169 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3. 1/1 2024-06-13T01:16:27.170 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:16:27.170 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-13T01:16:27.170 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:27.278 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:57 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:27.366 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:16:27.408 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:27.409 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:16:27.409 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:27.474 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-06-13T01:16:28.050 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:58 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3. 1/1 2024-06-13T01:16:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:28.078 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:28.078 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:28.078 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:28.177 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 3.4 M 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 24 k 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 15 k 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-13T01:16:28.178 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout:Install 7 Packages 2024-06-13T01:16:28.179 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:28.180 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.7 M 2024-06-13T01:16:28.180 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 80 M 2024-06-13T01:16:28.180 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:28.297 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2024-06-13T01:16:28.453 INFO:teuthology.orchestra.run.smithi155.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1006.g296 130 kB/s | 24 kB 00:00 2024-06-13T01:16:28.512 INFO:teuthology.orchestra.run.smithi155.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-06-13T01:16:28.545 INFO:teuthology.orchestra.run.smithi155.stdout:(3/7): ceph-prometheus-alerts-17.2.7-1006.g296b 52 kB/s | 15 kB 00:00 2024-06-13T01:16:28.571 INFO:teuthology.orchestra.run.smithi155.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 698 kB/s | 41 kB 00:00 2024-06-13T01:16:28.629 INFO:teuthology.orchestra.run.smithi155.stdout:(5/7): python3-routes-2.5.1-5.el9.noarch.rpm 5.3 MB/s | 188 kB 00:00 2024-06-13T01:16:28.654 INFO:teuthology.orchestra.run.smithi155.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 282 kB/s | 31 kB 00:00 2024-06-13T01:16:28.880 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:07 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:28.966 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:16:29.008 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:29.009 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:16:29.009 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:29.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2024-06-13T01:16:29.229 INFO:teuthology.orchestra.run.smithi155.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1006.g296b94f3 3.5 MB/s | 3.4 MB 00:00 2024-06-13T01:16:29.233 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:29.233 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.5 MB/s | 3.7 MB 00:01 2024-06-13T01:16:29.233 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:29.242 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:29.242 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:29.304 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:29.305 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:29.615 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:29.653 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:08 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:29.672 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-13T01:16:29.707 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-13T01:16:29.762 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-13T01:16:29.780 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 3.4 M 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 24 k 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 15 k 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-13T01:16:29.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout:Install 7 Packages 2024-06-13T01:16:29.782 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:29.783 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.7 M 2024-06-13T01:16:29.783 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 80 M 2024-06-13T01:16:29.783 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:29.802 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-13T01:16:29.831 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1006.g296b94f3.el9.n 5/7 2024-06-13T01:16:30.139 INFO:teuthology.orchestra.run.smithi132.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1006.g296b 75 kB/s | 15 kB 00:00 2024-06-13T01:16:30.200 INFO:teuthology.orchestra.run.smithi132.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-06-13T01:16:30.231 INFO:teuthology.orchestra.run.smithi132.stdout:(3/7): python3-jwt-2.4.0-1.el9.noarch.rpm 1.2 MB/s | 41 kB 00:00 2024-06-13T01:16:30.256 INFO:teuthology.orchestra.run.smithi132.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.2 MB/s | 31 kB 00:00 2024-06-13T01:16:30.298 INFO:teuthology.orchestra.run.smithi132.stdout:(5/7): ceph-grafana-dashboards-17.2.7-1006.g296 69 kB/s | 24 kB 00:00 2024-06-13T01:16:30.323 INFO:teuthology.orchestra.run.smithi132.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-06-13T01:16:30.557 INFO:teuthology.orchestra.run.smithi132.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1006.g296b94f3 5.5 MB/s | 3.4 MB 00:00 2024-06-13T01:16:30.560 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:30.561 INFO:teuthology.orchestra.run.smithi132.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2024-06-13T01:16:30.561 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:30.569 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:30.569 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:30.631 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:30.632 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:30.834 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1006.g296b94f3.el9. 6/7 2024-06-13T01:16:30.876 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarc 7/7 2024-06-13T01:16:30.931 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:31.000 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-13T01:16:31.034 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-13T01:16:31.089 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-13T01:16:31.129 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-13T01:16:31.158 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1006.g296b94f3.el9.n 5/7 2024-06-13T01:16:31.902 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarc 7/7 2024-06-13T01:16:31.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1006.g296b94f3.el9. 1/7 2024-06-13T01:16:31.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarc 2/7 2024-06-13T01:16:31.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1006.g296b94f3.el9.n 3/7 2024-06-13T01:16:31.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-13T01:16:31.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-13T01:16:31.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-13T01:16:32.156 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1006.g296b94f3.el9. 6/7 2024-06-13T01:16:32.204 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarc 7/7 2024-06-13T01:16:32.216 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:32.217 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:32.391 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-13T01:16:32.957 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:02 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:33.083 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:33.084 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 7.4 M 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-13T01:16:33.085 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-06-13T01:16:33.086 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:33.087 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 38 M 2024-06-13T01:16:33.087 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-06-13T01:16:33.087 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:33.364 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarc 7/7 2024-06-13T01:16:33.364 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1006.g296b94f3.el9. 1/7 2024-06-13T01:16:33.364 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarc 2/7 2024-06-13T01:16:33.364 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1006.g296b94f3.el9.n 3/7 2024-06-13T01:16:33.365 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-13T01:16:33.365 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-13T01:16:33.365 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-13T01:16:33.502 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:33.503 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:33.646 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 874 kB/s | 195 kB 00:00 2024-06-13T01:16:33.649 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:33.696 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-06-13T01:16:33.699 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:33.712 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-13T01:16:33.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-13T01:16:33.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-13T01:16:33.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-13T01:16:33.713 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:33.713 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:33.788 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-06-13T01:16:33.791 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:33.847 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 311 kB/s | 18 kB 00:00 2024-06-13T01:16:33.905 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 605 kB/s | 35 kB 00:00 2024-06-13T01:16:33.933 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-13T01:16:34.122 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-06-13T01:16:34.181 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-06-13T01:16:34.290 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-06-13T01:16:34.348 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-06-13T01:16:34.502 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:13 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:34.507 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-06-13T01:16:34.630 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:34.636 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-06-13T01:16:34.636 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:34.636 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-06-13T01:16:34.637 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:34.637 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 7.4 M 2024-06-13T01:16:34.637 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:16:34.637 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-13T01:16:34.637 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-13T01:16:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-13T01:16:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-13T01:16:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-13T01:16:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-13T01:16:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-13T01:16:34.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-13T01:16:34.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-13T01:16:34.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-13T01:16:34.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-13T01:16:34.639 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:34.640 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:34.640 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-06-13T01:16:34.640 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-06-13T01:16:34.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:34.640 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 38 M 2024-06-13T01:16:34.640 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 209 M 2024-06-13T01:16:34.641 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:35.007 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-06-13T01:16:35.099 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 4.4 MB/s | 7.4 MB 00:01 2024-06-13T01:16:35.101 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:35.101 INFO:teuthology.orchestra.run.smithi155.stdout:Total 19 MB/s | 38 MB 00:02 2024-06-13T01:16:35.109 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:35.110 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:35.283 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 772 kB/s | 195 kB 00:00 2024-06-13T01:16:35.285 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:35.325 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-06-13T01:16:35.328 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:35.453 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:35.480 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:35.480 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:35.551 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 122 kB/s | 33 kB 00:00 2024-06-13T01:16:35.553 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:35.659 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-06-13T01:16:35.759 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 353 kB/s | 35 kB 00:00 2024-06-13T01:16:35.960 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-10 8.0 MB/s | 7.4 MB 00:00 2024-06-13T01:16:35.989 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:35.989 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:36.435 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-06-13T01:16:36.470 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 480 kB/s | 244 kB 00:00 2024-06-13T01:16:36.669 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-06-13T01:16:36.749 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:36.832 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-13T01:16:36.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-13T01:16:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-13T01:16:36.977 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 5.1 MB 00:00 2024-06-13T01:16:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-13T01:16:37.190 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-13T01:16:37.222 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-13T01:16:37.310 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-13T01:16:37.444 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.8 MB/s | 4.7 MB 00:01 2024-06-13T01:16:37.958 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-13T01:16:38.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-13T01:16:38.278 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-06-13T01:16:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:Total 10 MB/s | 38 MB 00:03 2024-06-13T01:16:38.630 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:38.657 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:38.657 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:39.160 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:39.161 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-13T01:16:39.915 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:39.988 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-13T01:16:40.025 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-13T01:16:40.124 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-13T01:16:40.160 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-13T01:16:40.338 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-13T01:16:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-13T01:16:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-13T01:16:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b9 12/12 2024-06-13T01:16:40.454 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-13T01:16:41.102 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-13T01:16:41.214 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-13T01:16:42.459 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b9 12/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b9 1/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-13T01:16:42.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-13T01:16:42.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-13T01:16:42.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-13T01:16:42.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-13T01:16:43.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-13T01:16:43.261 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-13T01:16:43.262 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-13T01:16:43.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-13T01:16:43.263 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:43.263 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:43.503 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-13T01:16:43.534 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b9 12/12 2024-06-13T01:16:43.605 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-06-13T01:16:44.201 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:14 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:44.330 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 47 k 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:16:44.331 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-13T01:16:44.332 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.2 M 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-06-13T01:16:44.333 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:44.662 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:44.692 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:44.864 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): ceph-mgr-rook-17.2.7-1006.g296b94f3.el9 169 kB/s | 47 kB 00:00 2024-06-13T01:16:44.865 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:45.006 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 45 kB/s | 19 kB 00:00 2024-06-13T01:16:45.023 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 58 kB/s | 26 kB 00:00 2024-06-13T01:16:45.107 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 918 kB/s | 222 kB 00:00 2024-06-13T01:16:45.182 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 905 kB/s | 159 kB 00:00 2024-06-13T01:16:45.215 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 976 kB/s | 32 kB 00:00 2024-06-13T01:16:45.241 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 404 kB/s | 54 kB 00:00 2024-06-13T01:16:45.266 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 299 kB/s | 15 kB 00:00 2024-06-13T01:16:45.291 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-06-13T01:16:45.359 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 891 kB/s | 59 kB 00:00 2024-06-13T01:16:45.383 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-google-auth-2.29.0-2.el9.noarc 1.4 MB/s | 205 kB 00:00 2024-06-13T01:16:45.410 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-06-13T01:16:45.442 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-06-13T01:16:45.445 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:45.446 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-06-13T01:16:45.509 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:45.524 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:45.524 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:45.612 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:45.612 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:46.045 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b9 12/12 2024-06-13T01:16:46.045 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b9 1/12 2024-06-13T01:16:46.045 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-13T01:16:46.046 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-13T01:16:46.113 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-13T01:16:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-13T01:16:46.389 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-13T01:16:46.426 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-13T01:16:46.468 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-13T01:16:46.469 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-13T01:16:46.469 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:46.469 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:46.469 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:46.469 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-13T01:16:46.470 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-13T01:16:46.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-13T01:16:46.471 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:46.471 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:46.565 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-13T01:16:46.675 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-13T01:16:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-13T01:16:46.871 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2024-06-13T01:16:47.461 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:26 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:47.472 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-13T01:16:47.519 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-13T01:16:47.553 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-13T01:16:47.588 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 47 k 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-13T01:16:47.590 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-06-13T01:16:47.591 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:47.592 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.2 M 2024-06-13T01:16:47.592 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-06-13T01:16:47.592 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:47.599 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-13T01:16:47.623 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 13/13 2024-06-13T01:16:48.227 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:48.228 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 13/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 1/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-13T01:16:48.365 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-13T01:16:48.366 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-13T01:16:48.366 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-13T01:16:48.366 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-13T01:16:48.366 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-13T01:16:48.366 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-13T01:16:48.384 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 81 kB/s | 19 kB 00:00 2024-06-13T01:16:48.409 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 98 kB/s | 26 kB 00:00 2024-06-13T01:16:48.410 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:48.411 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:48.485 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): ceph-mgr-rook-17.2.7-1006.g296b94f3.el9 141 kB/s | 47 kB 00:00 2024-06-13T01:16:48.595 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 859 kB/s | 159 kB 00:00 2024-06-13T01:16:48.620 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 944 kB/s | 222 kB 00:00 2024-06-13T01:16:48.636 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-06-13T01:16:48.654 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 956 kB/s | 32 kB 00:00 2024-06-13T01:16:48.671 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 443 kB/s | 15 kB 00:00 2024-06-13T01:16:48.690 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-13T01:16:48.690 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:48.690 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:48.690 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:48.690 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-13T01:16:48.690 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:48.691 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:48.699 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-06-13T01:16:48.733 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-google-auth-2.29.0-2.el9.noarc 2.6 MB/s | 205 kB 00:00 2024-06-13T01:16:48.800 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-06-13T01:16:48.825 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 470 kB/s | 59 kB 00:00 2024-06-13T01:16:48.867 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.2 MB/s | 1.0 MB 00:00 2024-06-13T01:16:48.869 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:48.870 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-06-13T01:16:48.934 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:48.951 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:48.951 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:48.959 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-06-13T01:16:49.034 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:49.034 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:49.537 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:19 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:49.539 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:49.664 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 123 k 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-13T01:16:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 7.0 M 2024-06-13T01:16:49.666 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-06-13T01:16:49.667 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:49.725 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-13T01:16:49.786 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-13T01:16:49.841 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-13T01:16:49.878 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-13T01:16:49.920 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-13T01:16:50.023 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-13T01:16:50.133 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-13T01:16:50.180 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-13T01:16:50.183 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:50.184 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:50.363 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1006.g296b94f3.e 481 kB/s | 123 kB 00:00 2024-06-13T01:16:50.413 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-06-13T01:16:50.438 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 747 kB/s | 248 kB 00:00 2024-06-13T01:16:50.464 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-06-13T01:16:50.489 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-13T01:16:50.672 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-06-13T01:16:50.674 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:50.675 INFO:teuthology.orchestra.run.smithi155.stdout:Total 6.9 MB/s | 7.0 MB 00:01 2024-06-13T01:16:50.745 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:50.775 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:50.775 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:50.854 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:50.856 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:50.906 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-13T01:16:50.961 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-13T01:16:51.003 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-13T01:16:51.057 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-13T01:16:51.089 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 13/13 2024-06-13T01:16:51.159 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:51.324 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-13T01:16:51.374 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-13T01:16:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-13T01:16:51.921 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 13/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 1/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-13T01:16:51.922 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-13T01:16:51.923 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-13T01:16:51.923 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-13T01:16:51.923 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-13T01:16:51.923 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-13T01:16:51.994 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-13T01:16:52.050 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-13T01:16:52.078 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 6/6 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-13T01:16:52.243 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:52.244 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:52.491 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2024-06-13T01:16:52.802 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 6/6 2024-06-13T01:16:52.802 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/6 2024-06-13T01:16:52.802 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-13T01:16:52.802 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-13T01:16:52.802 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-13T01:16:52.802 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-13T01:16:53.062 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:32 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:53.072 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-13T01:16:53.072 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:53.072 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:53.072 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:53.073 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:53.189 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:53.189 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1006.g296b94f3.el9 ceph-noarch 123 k 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-13T01:16:53.190 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-13T01:16:53.191 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-13T01:16:53.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:53.191 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:53.191 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:53.191 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-06-13T01:16:53.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:53.192 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 7.0 M 2024-06-13T01:16:53.192 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 33 M 2024-06-13T01:16:53.192 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:53.304 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-06-13T01:16:53.366 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:53.368 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-13T01:16:53.620 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1006.g296b94f3.e 371 kB/s | 123 kB 00:00 2024-06-13T01:16:53.754 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.3 MB/s | 548 kB 00:00 2024-06-13T01:16:53.779 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.4 MB/s | 58 kB 00:00 2024-06-13T01:16:53.804 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-13T01:16:53.875 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:23 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:54.003 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:54.004 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:54.004 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-06-13T01:16:54.004 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:54.004 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 812 k 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-06-13T01:16:54.005 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:54.006 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 893 k 2024-06-13T01:16:54.006 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.6 M 2024-06-13T01:16:54.006 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:54.111 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:54.258 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 383 kB/s | 80 kB 00:00 2024-06-13T01:16:54.288 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 248 kB/s | 248 kB 00:00 2024-06-13T01:16:54.467 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-17.2.7-1006.g296b94f3.el9.x86_ 1.9 MB/s | 812 kB 00:00 2024-06-13T01:16:54.468 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:54.469 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.9 MB/s | 893 kB 00:00 2024-06-13T01:16:54.480 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:54.503 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:54.503 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:54.581 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:54.582 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:54.721 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-06-13T01:16:54.724 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:54.724 INFO:teuthology.orchestra.run.smithi132.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-06-13T01:16:54.785 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:54.794 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:54.806 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:54.806 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:54.902 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:54.903 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:54.912 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-13T01:16:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 2/2 2024-06-13T01:16:55.211 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:55.377 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-13T01:16:55.431 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-13T01:16:55.639 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 2/2 2024-06-13T01:16:55.639 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/2 2024-06-13T01:16:55.922 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-13T01:16:55.922 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:55.923 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:55.923 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-13T01:16:55.923 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:55.923 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:55.960 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-13T01:16:56.042 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-13T01:16:56.107 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-13T01:16:56.136 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 6/6 2024-06-13T01:16:56.229 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-06-13T01:16:56.814 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:26 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:56.925 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 6/6 2024-06-13T01:16:56.925 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 1/6 2024-06-13T01:16:56.926 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-13T01:16:56.926 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-13T01:16:56.926 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-13T01:16:56.926 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 126 k 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:16:56.946 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:16:56.947 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:16:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:56.947 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 126 k 2024-06-13T01:16:56.947 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 448 k 2024-06-13T01:16:56.947 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:16:57.191 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-13T01:16:57.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:17.2.7-1006.g296b94f3.el9.noarch 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:57.192 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:16:57.264 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-17.2.7-1006.g296b94f3.el9.x86_64 398 kB/s | 126 kB 00:00 2024-06-13T01:16:57.265 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:57.265 INFO:teuthology.orchestra.run.smithi155.stdout:Total 396 kB/s | 126 kB 00:00 2024-06-13T01:16:57.265 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:16:57.271 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:16:57.271 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:16:57.332 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:16:57.333 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:16:57.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2024-06-13T01:16:57.483 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:16:57.586 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:16:58.029 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:37 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:16:58.041 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:16:58.158 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 812 k 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-13T01:16:58.159 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 893 k 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.6 M 2024-06-13T01:16:58.160 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:16:58.313 INFO:teuthology.orchestra.run.smithi132.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-13T01:16:58.322 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:16:58.322 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:58.322 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:16:58.322 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:16:58.323 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:16:58.323 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:58.490 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 377 kB/s | 80 kB 00:00 2024-06-13T01:16:58.516 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-06-13T01:16:58.699 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-fuse-17.2.7-1006.g296b94f3.el9.x86_ 1.9 MB/s | 812 kB 00:00 2024-06-13T01:16:58.700 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:16:58.700 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-06-13T01:16:58.712 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:16:58.734 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:16:58.735 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:16:58.813 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:16:58.814 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:16:59.002 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:16:59.109 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:29 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:59.145 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-13T01:16:59.197 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:16:59.223 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 2/2 2024-06-13T01:16:59.241 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:16:59.242 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:16:59.242 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:16:59.301 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-06-13T01:16:59.874 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:29 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:16:59.906 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 2/2 2024-06-13T01:16:59.906 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/2 2024-06-13T01:17:00.000 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 25 k 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 25 k 2024-06-13T01:17:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 110 k 2024-06-13T01:17:00.002 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:17:00.185 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-13T01:17:00.185 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:00.185 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:17:00.185 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-13T01:17:00.185 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:00.185 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:00.244 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-17.2.7-1006.g296b94f3.el9.x86_6 103 kB/s | 25 kB 00:00 2024-06-13T01:17:00.244 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:00.245 INFO:teuthology.orchestra.run.smithi155.stdout:Total 102 kB/s | 25 kB 00:00 2024-06-13T01:17:00.245 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:17:00.248 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:17:00.248 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:17:00.250 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:17:00.250 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:17:00.331 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:17:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:00.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2024-06-13T01:17:00.826 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:01.023 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:40 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:01.063 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:01.063 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:01.063 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:17:01.064 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:01.064 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:01.064 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:01.152 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:01.153 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:01.153 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-06-13T01:17:01.153 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:01.153 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:17:01.153 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 126 k 2024-06-13T01:17:01.153 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 126 k 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 448 k 2024-06-13T01:17:01.154 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:17:01.237 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-06-13T01:17:01.379 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-17.2.7-1006.g296b94f3.el9.x86_64 561 kB/s | 126 kB 00:00 2024-06-13T01:17:01.380 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:01.380 INFO:teuthology.orchestra.run.smithi132.stdout:Total 557 kB/s | 126 kB 00:00 2024-06-13T01:17:01.380 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:17:01.386 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:17:01.386 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:17:01.447 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:17:01.448 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:17:01.608 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:17:01.701 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:01.806 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:31 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:01.890 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:01.931 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:01.932 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:17:01.933 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:02.000 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-06-13T01:17:02.134 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:02.397 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:02.397 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:02.397 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:17:02.397 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:02.397 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:02.397 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:02.568 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:32 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:02.640 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2024-06-13T01:17:02.652 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:02.694 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:02.695 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:17:02.695 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:02.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-06-13T01:17:03.242 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:42 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:03.331 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:03.339 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:33 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:03.374 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:03.375 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:03.375 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:03.423 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:03.440 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2024-06-13T01:17:03.465 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:03.466 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:17:03.466 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:03.534 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-06-13T01:17:04.035 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:43 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:04.111 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:34 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:04.160 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 25 k 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:04.161 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:17:04.162 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:04.162 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:17:04.162 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:04.162 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 25 k 2024-06-13T01:17:04.162 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 110 k 2024-06-13T01:17:04.162 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:17:04.195 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:04.236 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:04.237 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:17:04.237 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:04.300 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-06-13T01:17:04.401 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-17.2.7-1006.g296b94f3.el9.x86_6 104 kB/s | 25 kB 00:00 2024-06-13T01:17:04.402 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:04.402 INFO:teuthology.orchestra.run.smithi132.stdout:Total 104 kB/s | 25 kB 00:00 2024-06-13T01:17:04.402 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:17:04.405 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:17:04.405 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:17:04.407 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:17:04.407 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:17:04.491 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:17:04.597 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:04.874 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:34 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:04.959 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:04.959 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:05.000 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:05.001 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:17:05.001 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:05.067 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-06-13T01:17:05.180 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:05.180 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:05.180 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:17:05.180 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:05.181 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:05.181 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:05.390 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2024-06-13T01:17:05.638 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:35 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:05.722 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:05.765 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:05.766 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-06-13T01:17:05.766 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:05.825 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-06-13T01:17:05.962 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:44 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:06.046 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:06.087 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:06.089 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:06.089 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:06.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2024-06-13T01:17:06.395 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:36 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:06.520 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:06.521 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:06.521 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 82 k 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 82 k 2024-06-13T01:17:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 222 k 2024-06-13T01:17:06.523 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:17:06.722 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-17.2.7-1006.g296b94f3.el9.x86_64.rpm 408 kB/s | 82 kB 00:00 2024-06-13T01:17:06.723 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:06.723 INFO:teuthology.orchestra.run.smithi155.stdout:Total 406 kB/s | 82 kB 00:00 2024-06-13T01:17:06.723 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:17:06.726 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:45 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:06.730 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:17:06.730 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:17:06.791 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:17:06.792 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:17:06.811 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:06.852 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:06.853 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:06.853 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:06.923 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2024-06-13T01:17:06.934 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:17:07.036 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:07.401 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:07.499 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:46 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:07.583 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:07.625 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:07.626 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:07.626 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:07.680 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:07.680 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:07.680 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:17:07.680 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:07.680 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:07.680 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:07.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2024-06-13T01:17:07.907 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-06-13T01:17:08.257 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:47 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:08.341 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:08.382 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:08.383 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:08.383 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:08.445 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2024-06-13T01:17:08.501 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:38 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:08.630 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 2.9 M 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:17:08.631 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:08.632 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:17:08.632 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:08.632 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.9 M 2024-06-13T01:17:08.632 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 12 M 2024-06-13T01:17:08.632 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:17:09.011 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:48 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:09.094 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:09.135 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:09.136 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:09.136 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:09.168 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-17.2.7-1006.g296b94f3.el9.x86_64.rpm 5.5 MB/s | 2.9 MB 00:00 2024-06-13T01:17:09.168 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:09.169 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.5 MB/s | 2.9 MB 00:00 2024-06-13T01:17:09.169 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:17:09.178 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:17:09.178 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:17:09.194 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2024-06-13T01:17:09.256 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:17:09.256 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:17:09.662 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:17:09.710 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:09.734 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:09.735 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:17:09.735 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-13T01:17:09.735 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T01:17:09.735 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T01:17:09.735 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:09.766 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:48 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:09.853 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:17.2.7-1006.g296b94f3.el9.x86_64 is already installed. 2024-06-13T01:17:09.892 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:09.893 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-06-13T01:17:09.893 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:09.956 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2024-06-13T01:17:10.521 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:49 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:10.646 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:10.647 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:10.647 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-06-13T01:17:10.647 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:10.647 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:17:10.647 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 82 k 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 82 k 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 222 k 2024-06-13T01:17:10.648 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:17:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:10.829 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:17:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:10.829 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:10.829 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:10.854 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-17.2.7-1006.g296b94f3.el9.x86_64.rpm 395 kB/s | 82 kB 00:00 2024-06-13T01:17:10.855 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:10.855 INFO:teuthology.orchestra.run.smithi132.stdout:Total 393 kB/s | 82 kB 00:00 2024-06-13T01:17:10.856 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:17:10.862 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:17:10.862 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:17:10.923 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:17:10.923 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:17:11.038 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-06-13T01:17:11.068 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:17:11.167 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:11.544 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:11.708 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:41 ago on Thu 13 Jun 2024 01:15:30 AM UTC. 2024-06-13T01:17:11.751 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-06-13T01:17:11.751 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:11.751 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 165 k 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 165 k 2024-06-13T01:17:11.752 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 477 k 2024-06-13T01:17:11.753 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-06-13T01:17:11.821 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:11.821 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:11.821 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:17:11.821 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:11.821 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:11.821 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:11.989 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-17.2.7-1006.g296b94f3.el9.x86_64.rpm 698 kB/s | 165 kB 00:00 2024-06-13T01:17:11.990 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:11.990 INFO:teuthology.orchestra.run.smithi155.stdout:Total 693 kB/s | 165 kB 00:00 2024-06-13T01:17:11.990 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-06-13T01:17:11.997 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-06-13T01:17:11.997 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-06-13T01:17:12.059 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-06-13T01:17:12.059 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-06-13T01:17:12.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-06-13T01:17:12.218 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-06-13T01:17:12.321 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:12.653 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:51 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:12.756 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:12.784 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 2.9 M 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:17:12.785 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:12.786 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:17:12.786 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:12.786 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.9 M 2024-06-13T01:17:12.786 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 12 M 2024-06-13T01:17:12.786 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:17:13.060 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:13.060 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:13.060 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-06-13T01:17:13.060 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:13.061 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:17:13.061 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-06-13T01:17:13.296 DEBUG:teuthology.parallel:result is None 2024-06-13T01:17:13.317 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-17.2.7-1006.g296b94f3.el9.x86_64.rpm 5.5 MB/s | 2.9 MB 00:00 2024-06-13T01:17:13.318 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:13.318 INFO:teuthology.orchestra.run.smithi132.stdout:Total 5.5 MB/s | 2.9 MB 00:00 2024-06-13T01:17:13.319 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:17:13.333 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:17:13.333 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:17:13.420 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:17:13.420 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:17:13.833 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:17:13.884 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:13.910 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:13.910 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-13T01:17:13.911 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-13T01:17:13.911 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T01:17:13.911 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-13T01:17:13.911 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:14.963 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:14.963 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:14.963 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:17:14.964 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:14.964 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:14.964 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:15.212 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2024-06-13T01:17:15.798 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:54 ago on Thu 13 Jun 2024 01:15:21 AM UTC. 2024-06-13T01:17:15.927 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:17.2.7-1006.g296b94f3.el9 ceph 165 k 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:15.928 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-06-13T01:17:15.929 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-06-13T01:17:15.929 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-06-13T01:17:15.929 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:15.929 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 165 k 2024-06-13T01:17:15.929 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 477 k 2024-06-13T01:17:15.929 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-06-13T01:17:16.279 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-17.2.7-1006.g296b94f3.el9.x86_64.rpm 471 kB/s | 165 kB 00:00 2024-06-13T01:17:16.280 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:17:16.280 INFO:teuthology.orchestra.run.smithi132.stdout:Total 470 kB/s | 165 kB 00:00 2024-06-13T01:17:16.280 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-06-13T01:17:16.287 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-06-13T01:17:16.288 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-06-13T01:17:16.350 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-06-13T01:17:16.350 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-06-13T01:17:16.511 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-06-13T01:17:16.618 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:17.044 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:17.287 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 1/1 2024-06-13T01:17:17.287 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:17.287 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-06-13T01:17:17.287 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:17.2.7-1006.g296b94f3.el9.x86_64 2024-06-13T01:17:17.288 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:17:17.288 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-06-13T01:17:17.536 DEBUG:teuthology.parallel:result is None 2024-06-13T01:17:17.536 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:17:17.536 INFO:teuthology.packaging:ref: None 2024-06-13T01:17:17.536 INFO:teuthology.packaging:tag: None 2024-06-13T01:17:17.536 INFO:teuthology.packaging:branch: quincy 2024-06-13T01:17:17.536 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:17:17.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-13T01:17:17.748 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-13T01:17:17.769 INFO:teuthology.orchestra.run.smithi132.stdout:17.2.7-1006.g296b94f3.el9 2024-06-13T01:17:17.770 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1006.g296b94f3.el9 2024-06-13T01:17:17.770 INFO:teuthology.task.install:The correct ceph version 17.2.7-1006.g296b94f3 is installed. 2024-06-13T01:17:17.771 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-13T01:17:17.771 INFO:teuthology.packaging:ref: None 2024-06-13T01:17:17.772 INFO:teuthology.packaging:tag: None 2024-06-13T01:17:17.772 INFO:teuthology.packaging:branch: quincy 2024-06-13T01:17:17.772 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-13T01:17:17.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-13T01:17:18.063 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-13T01:17:18.090 INFO:teuthology.orchestra.run.smithi155.stdout:17.2.7-1006.g296b94f3.el9 2024-06-13T01:17:18.090 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1006.g296b94f3.el9 2024-06-13T01:17:18.091 INFO:teuthology.task.install:The correct ceph version 17.2.7-1006.g296b94f3 is installed. 2024-06-13T01:17:18.092 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-13T01:17:18.092 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:17:18.092 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-13T01:17:18.119 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:17:18.119 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-13T01:17:18.160 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-13T01:17:18.160 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:17:18.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-06-13T01:17:18.186 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-13T01:17:18.252 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:17:18.252 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-06-13T01:17:18.281 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-13T01:17:18.349 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-13T01:17:18.349 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:17:18.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-13T01:17:18.375 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-13T01:17:18.441 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:17:18.441 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-13T01:17:18.468 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-13T01:17:18.534 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-13T01:17:18.534 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:17:18.534 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-06-13T01:17:18.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-13T01:17:18.625 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:17:18.625 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-06-13T01:17:18.651 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-13T01:17:18.717 INFO:teuthology.run_tasks:Running task print... 2024-06-13T01:17:18.726 INFO:teuthology.task.print:**** done install task... 2024-06-13T01:17:18.726 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-13T01:17:18.911 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-13T01:17:18.911 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-13T01:17:18.912 INFO:tasks.cephadm:Cluster fsid is ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:17:18.912 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-13T01:17:18.912 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-13T01:17:18.912 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi132': '172.21.15.132', 'mon.smithi155': '172.21.15.155'} 2024-06-13T01:17:18.912 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-13T01:17:18.912 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-06-13T01:17:18.937 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-06-13T01:17:18.964 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-13T01:17:18.964 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-13T01:17:19.286 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 13 01:17 /home/ubuntu/cephtest/cephadm 2024-06-13T01:17:19.286 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-13T01:17:19.514 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 13 01:17 /home/ubuntu/cephtest/cephadm 2024-06-13T01:17:19.515 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-13T01:17:19.532 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-13T01:17:19.557 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-13T01:17:19.558 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-13T01:17:19.574 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-13T01:17:20.430 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-13T01:17:20.440 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-13T01:18:02.843 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-06-13T01:18:02.844 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 17.2.7-1006-g296b94f3 (296b94f31805b13f169a9221a78e5c5d2902e4d0) quincy (stable)", 2024-06-13T01:18:02.844 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a", 2024-06-13T01:18:02.844 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-06-13T01:18:02.844 INFO:teuthology.orchestra.run.smithi132.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56" 2024-06-13T01:18:02.844 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-06-13T01:18:02.844 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-06-13T01:18:08.035 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-06-13T01:18:08.035 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 17.2.7-1006-g296b94f3 (296b94f31805b13f169a9221a78e5c5d2902e4d0) quincy (stable)", 2024-06-13T01:18:08.035 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a", 2024-06-13T01:18:08.035 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2024-06-13T01:18:08.036 INFO:teuthology.orchestra.run.smithi155.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56" 2024-06-13T01:18:08.036 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-06-13T01:18:08.036 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-06-13T01:18:08.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-06-13T01:18:08.088 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-06-13T01:18:08.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-06-13T01:18:08.156 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-06-13T01:18:08.180 INFO:tasks.cephadm:Writing seed config... 2024-06-13T01:18:08.181 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-13T01:18:08.181 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-13T01:18:08.182 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-13T01:18:08.183 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-13T01:18:08.185 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-13T01:18:08.186 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-13T01:18:08.187 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-13T01:18:08.188 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:18:08.188 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-13T01:18:08.236 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ad8d4854-2922-11ef-bc9c-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-13T01:18:08.237 DEBUG:teuthology.orchestra.run.smithi132:mon.smithi132> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service 2024-06-13T01:18:08.279 INFO:tasks.cephadm:Bootstrapping... 2024-06-13T01:18:08.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.132 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:18:08.497 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-06-13T01:18:08.497 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'ad8d4854-2922-11ef-bc9c-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.132', '--skip-admin-label'] 2024-06-13T01:18:08.524 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stdout 5.1.0 2024-06-13T01:18:08.524 INFO:teuthology.orchestra.run.smithi132.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-13T01:18:08.524 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying podman|docker is present... 2024-06-13T01:18:08.548 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stdout 5.1.0 2024-06-13T01:18:08.549 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying lvm2 is present... 2024-06-13T01:18:08.549 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying time synchronization is in place... 2024-06-13T01:18:08.557 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-13T01:18:08.557 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-13T01:18:08.564 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-13T01:18:08.564 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-06-13T01:18:08.571 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout enabled 2024-06-13T01:18:08.578 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout active 2024-06-13T01:18:08.578 INFO:teuthology.orchestra.run.smithi132.stdout:Unit chronyd.service is enabled and running 2024-06-13T01:18:08.578 INFO:teuthology.orchestra.run.smithi132.stdout:Repeating the final host check... 2024-06-13T01:18:08.602 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stdout 5.1.0 2024-06-13T01:18:08.602 INFO:teuthology.orchestra.run.smithi132.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-13T01:18:08.602 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl is present 2024-06-13T01:18:08.602 INFO:teuthology.orchestra.run.smithi132.stdout:lvcreate is present 2024-06-13T01:18:08.609 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-13T01:18:08.609 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-13T01:18:08.616 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-13T01:18:08.616 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-06-13T01:18:08.622 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout enabled 2024-06-13T01:18:08.629 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout active 2024-06-13T01:18:08.629 INFO:teuthology.orchestra.run.smithi132.stdout:Unit chronyd.service is enabled and running 2024-06-13T01:18:08.629 INFO:teuthology.orchestra.run.smithi132.stdout:Host looks OK 2024-06-13T01:18:08.629 INFO:teuthology.orchestra.run.smithi132.stdout:Cluster fsid: ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:08.629 INFO:teuthology.orchestra.run.smithi132.stdout:Acquiring lock 140327343141312 on /run/cephadm/ad8d4854-2922-11ef-bc9c-c7b262605968.lock 2024-06-13T01:18:08.629 INFO:teuthology.orchestra.run.smithi132.stdout:Lock 140327343141312 acquired on /run/cephadm/ad8d4854-2922-11ef-bc9c-c7b262605968.lock 2024-06-13T01:18:08.630 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying IP 172.21.15.132 port 3300 ... 2024-06-13T01:18:08.630 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying IP 172.21.15.132 port 6789 ... 2024-06-13T01:18:08.630 INFO:teuthology.orchestra.run.smithi132.stdout:Base mon IP(s) is [172.21.15.132:3300, 172.21.15.132:6789], mon addrv is [v2:172.21.15.132:3300,v1:172.21.15.132:6789] 2024-06-13T01:18:08.634 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.132 metric 100 2024-06-13T01:18:08.634 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.132 metric 100 2024-06-13T01:18:08.637 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-13T01:18:08.637 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-13T01:18:08.637 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1784sec hoplimit 64 pref medium 2024-06-13T01:18:08.639 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-13T01:18:08.639 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:773f/64 scope link 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.0/20` 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.0/20` 2024-06-13T01:18:08.640 INFO:teuthology.orchestra.run.smithi132.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-13T01:18:08.641 INFO:teuthology.orchestra.run.smithi132.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-13T01:18:08.642 INFO:teuthology.orchestra.run.smithi132.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-13T01:18:12.098 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stdout 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a 2024-06-13T01:18:12.099 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-13T01:18:12.099 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stderr Getting image source signatures 2024-06-13T01:18:12.099 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stderr Copying blob sha256:4da23cd526a9ade1f14fa735e40938f7b32896dd25f49903802d70720ee8debc 2024-06-13T01:18:12.099 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stderr Copying blob sha256:9065cd298533077f0006921bb12bd331d24f6c140cc8fa972c2e6c4e382038d1 2024-06-13T01:18:12.099 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stderr Copying config sha256:85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a 2024-06-13T01:18:12.099 INFO:teuthology.orchestra.run.smithi132.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-13T01:18:19.199 INFO:teuthology.orchestra.run.smithi132.stdout:ceph: stdout ceph version 17.2.7-1006-g296b94f3 (296b94f31805b13f169a9221a78e5c5d2902e4d0) quincy (stable) 2024-06-13T01:18:19.199 INFO:teuthology.orchestra.run.smithi132.stdout:Ceph version: ceph version 17.2.7-1006-g296b94f3 (296b94f31805b13f169a9221a78e5c5d2902e4d0) quincy (stable) 2024-06-13T01:18:19.199 INFO:teuthology.orchestra.run.smithi132.stdout:Extracting ceph user uid/gid from container image... 2024-06-13T01:18:20.130 INFO:teuthology.orchestra.run.smithi132.stdout:stat: stdout 167 167 2024-06-13T01:18:20.130 INFO:teuthology.orchestra.run.smithi132.stdout:Creating initial keys... 2024-06-13T01:18:21.076 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-authtool: stdout AQDcSGpmbmUqHhAAc8+WtfJs8Rj+9INydSjLQg== 2024-06-13T01:18:21.999 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-authtool: stdout AQDdSGpmXP79FxAA9GB6o8K9pVlSIk0f4i71AQ== 2024-06-13T01:18:22.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-authtool: stdout AQDeSGpm/bjXExAAjQJ7ccC4Erfht0Thry4CWg== 2024-06-13T01:18:22.874 INFO:teuthology.orchestra.run.smithi132.stdout:Creating initial monmap... 2024-06-13T01:18:23.721 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-13T01:18:23.721 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool for smithi132 [v2:172.21.15.132:3300,v1:172.21.15.132:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:setting min_mon_release = octopus 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: set fsid to ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:18:23.722 INFO:teuthology.orchestra.run.smithi132.stdout:Creating mon... 2024-06-13T01:18:25.432 INFO:teuthology.orchestra.run.smithi132.stdout:create mon.smithi132 on 2024-06-13T01:18:25.636 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-13T01:18:25.798 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-13T01:18:25.953 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968.target → /etc/systemd/system/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968.target. 2024-06-13T01:18:25.953 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968.target → /etc/systemd/system/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968.target. 2024-06-13T01:18:26.133 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132 2024-06-13T01:18:26.134 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service: Unit ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service not loaded. 2024-06-13T01:18:26.298 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968.target.wants/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service → /etc/systemd/system/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@.service. 2024-06-13T01:18:26.865 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T01:18:26.865 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout disabled 2024-06-13T01:18:26.873 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T01:18:26.873 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-06-13T01:18:26.873 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld.service is not enabled 2024-06-13T01:18:26.873 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to enable service . firewalld.service is not available 2024-06-13T01:18:26.874 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mon to start... 2024-06-13T01:18:26.874 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mon... 2024-06-13T01:18:26.928 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:26 smithi132 systemd[1]: Started Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968. 2024-06-13T01:18:27.982 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout cluster: 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout id: ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout services: 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi132 (age 0.621093s) 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout data: 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-13T01:18:27.983 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout pgs: 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.337+0000 7f036f616640 1 Processor -- start 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.337+0000 7f036f616640 1 -- start start 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.338+0000 7f036f616640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.338+0000 7f036f616640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681015a0 con 0x7f0368100c50 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.338+0000 7f036d3b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.338+0000 7f036d3b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51372/0 (socket says 172.21.15.132:51372) 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.338+0000 7f036d3b2640 1 -- 172.21.15.132:0/3782631074 learned_addr learned my addr 172.21.15.132:0/3782631074 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.338+0000 7f036d3b2640 1 -- 172.21.15.132:0/3782631074 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03681016e0 con 0x7f0368100c50 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.340+0000 7f036d3b2640 1 --2- 172.21.15.132:0/3782631074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368101060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0360009e50 tx=0x7f0360009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d781a231a192fa6 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:27.984 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f0367fff640 1 -- 172.21.15.132:0/3782631074 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360031560 con 0x7f0368100c50 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f0367fff640 1 -- 172.21.15.132:0/3782631074 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036002d050 con 0x7f0368100c50 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f0367fff640 1 -- 172.21.15.132:0/3782631074 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360031880 con 0x7f0368100c50 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 -- 172.21.15.132:0/3782631074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 msgr2=0x7f0368101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 --2- 172.21.15.132:0/3782631074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368101060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0360009e50 tx=0x7f0360009840 comp rx=0 tx=0).stop 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 -- 172.21.15.132:0/3782631074 shutdown_connections 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 --2- 172.21.15.132:0/3782631074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368101060 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 -- 172.21.15.132:0/3782631074 >> 172.21.15.132:0/3782631074 conn(0x7f03680fc430 msgr2=0x7f03680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 -- 172.21.15.132:0/3782631074 shutdown_connections 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.341+0000 7f036f616640 1 -- 172.21.15.132:0/3782631074 wait complete. 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036f616640 1 Processor -- start 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036f616640 1 -- start start 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036f616640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:27.985 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036f616640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681130b0 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036d3b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036d3b2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368075af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51384/0 (socket says 172.21.15.132:51384) 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036d3b2640 1 -- 172.21.15.132:0/2368794793 learned_addr learned my addr 172.21.15.132:0/2368794793 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036d3b2640 1 -- 172.21.15.132:0/2368794793 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360009ae0 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.342+0000 7f036d3b2640 1 --2- 172.21.15.132:0/2368794793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368075af0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f036002abe0 tx=0x7f0360036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360007a10 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360007b70 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03681131f0 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360007e80 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0368113660 con 0x7f0368100c50 2024-06-13T01:18:27.986 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0360038070 con 0x7f0368100c50 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.343+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0360045bc0 con 0x7f0368100c50 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.344+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036804b480 con 0x7f0368100c50 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.346+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f0360040070 con 0x7f0368100c50 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.408+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f036802a8e0 con 0x7f0368100c50 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.408+0000 7f0365ffb640 1 -- 172.21.15.132:0/2368794793 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f0360036070 con 0x7f0368100c50 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 msgr2=0x7f0368075af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 --2- 172.21.15.132:0/2368794793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368075af0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f036002abe0 tx=0x7f0360036000 comp rx=0 tx=0).stop 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 shutdown_connections 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 --2- 172.21.15.132:0/2368794793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0368100c50 0x7f0368075af0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 >> 172.21.15.132:0/2368794793 conn(0x7f03680fc430 msgr2=0x7f0368106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 shutdown_connections 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:27.409+0000 7f036f616640 1 -- 172.21.15.132:0/2368794793 wait complete. 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:mon is available 2024-06-13T01:18:27.987 INFO:teuthology.orchestra.run.smithi132.stdout:Assimilating anything we can from ceph.conf... 2024-06-13T01:18:28.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:27 smithi132 ceph-mon[28961]: mon.smithi132 is new leader, mons smithi132 in quorum (ranks 0) 2024-06-13T01:18:28.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:27 smithi132 ceph-mon[28961]: monmap e1: 1 mons at {smithi132=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-13T01:18:28.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:27 smithi132 ceph-mon[28961]: fsmap 2024-06-13T01:18:28.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:27 smithi132 ceph-mon[28961]: osdmap e1: 0 total, 0 up, 0 in 2024-06-13T01:18:28.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:27 smithi132 ceph-mon[28961]: mgrmap e1: no daemons active 2024-06-13T01:18:28.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:27 smithi132 ceph-mon[28961]: from='client.? 172.21.15.132:0/2368794793' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-13T01:18:28.899 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [global] 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout fsid = ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.132:3300,v1:172.21.15.132:6789] 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:28.906 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [osd] 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda513a5640 1 Processor -- start 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda513a5640 1 -- start start 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda513a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda513a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c075f30 con 0x7fda4c075b20 2024-06-13T01:18:28.907 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda4affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda4affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51400/0 (socket says 172.21.15.132:51400) 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.370+0000 7fda4affd640 1 -- 172.21.15.132:0/3320262886 learned_addr learned my addr 172.21.15.132:0/3320262886 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.371+0000 7fda4affd640 1 -- 172.21.15.132:0/3320262886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c104f20 con 0x7fda4c075b20 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.371+0000 7fda4affd640 1 --2- 172.21.15.132:0/3320262886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c1049e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009840 tx=0x7fda3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390d15a6afdd7518 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda4a7fc640 1 -- 172.21.15.132:0/3320262886 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda3c032440 con 0x7fda4c075b20 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda4a7fc640 1 -- 172.21.15.132:0/3320262886 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c0325a0 con 0x7fda4c075b20 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda4a7fc640 1 -- 172.21.15.132:0/3320262886 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda3c0328d0 con 0x7fda4c075b20 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 -- 172.21.15.132:0/3320262886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 msgr2=0x7fda4c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 --2- 172.21.15.132:0/3320262886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c1049e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009840 tx=0x7fda3c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 -- 172.21.15.132:0/3320262886 shutdown_connections 2024-06-13T01:18:28.908 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 --2- 172.21.15.132:0/3320262886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c1049e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 -- 172.21.15.132:0/3320262886 >> 172.21.15.132:0/3320262886 conn(0x7fda4c0fc470 msgr2=0x7fda4c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 -- 172.21.15.132:0/3320262886 shutdown_connections 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.372+0000 7fda513a5640 1 -- 172.21.15.132:0/3320262886 wait complete. 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda513a5640 1 Processor -- start 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda513a5640 1 -- start start 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda513a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c110330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda513a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c110870 con 0x7fda4c075b20 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda4affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c110330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda4affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c110330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51414/0 (socket says 172.21.15.132:51414) 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda4affd640 1 -- 172.21.15.132:0/1462387001 learned_addr learned my addr 172.21.15.132:0/1462387001 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.373+0000 7fda4affd640 1 -- 172.21.15.132:0/1462387001 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c009450 con 0x7fda4c075b20 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda4affd640 1 --2- 172.21.15.132:0/1462387001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c110330 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fda3c02a700 tx=0x7fda3c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:28.909 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda3c031be0 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c031d40 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c110a70 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda3c03b5a0 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c110e80 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fda3c039040 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.374+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fda3c041030 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.375+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c04b480 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.377+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fda3c037070 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.435+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fda4c109ae0 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.439+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda3c046d00 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.439+0000 7fda43fff640 1 -- 172.21.15.132:0/1462387001 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fda3c04eca0 con 0x7fda4c075b20 2024-06-13T01:18:28.910 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 msgr2=0x7fda4c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 --2- 172.21.15.132:0/1462387001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c110330 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fda3c02a700 tx=0x7fda3c032000 comp rx=0 tx=0).stop 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 shutdown_connections 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 --2- 172.21.15.132:0/1462387001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda4c075b20 0x7fda4c110330 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 >> 172.21.15.132:0/1462387001 conn(0x7fda4c0fc470 msgr2=0x7fda4c0fc840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 shutdown_connections 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:28.440+0000 7fda513a5640 1 -- 172.21.15.132:0/1462387001 wait complete. 2024-06-13T01:18:28.911 INFO:teuthology.orchestra.run.smithi132.stdout:Generating new minimal ceph.conf... 2024-06-13T01:18:28.959 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:28 smithi132 ceph-mon[28961]: from='client.? 172.21.15.132:0/1462387001' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-13T01:18:28.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:28 smithi132 ceph-mon[28961]: from='client.? 172.21.15.132:0/1462387001' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-13T01:18:29.959 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:29 smithi132 ceph-mon[28961]: from='client.? 172.21.15.132:0/1291444260' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.283+0000 7fd84f00e640 1 Processor -- start 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.283+0000 7fd84f00e640 1 -- start start 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.284+0000 7fd84f00e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd8481021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.284+0000 7fd84f00e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848102700 con 0x7fd848101db0 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.284+0000 7fd84cdaa640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd8481021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.284+0000 7fd84cdaa640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd8481021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51430/0 (socket says 172.21.15.132:51430) 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.284+0000 7fd84cdaa640 1 -- 172.21.15.132:0/2104853614 learned_addr learned my addr 172.21.15.132:0/2104853614 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:30.864 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.284+0000 7fd84cdaa640 1 -- 172.21.15.132:0/2104853614 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848102840 con 0x7fd848101db0 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.285+0000 7fd84cdaa640 1 --2- 172.21.15.132:0/2104853614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd8481021c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd844009840 tx=0x7fd8440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5870ab97389c3b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.285+0000 7fd83f7fe640 1 -- 172.21.15.132:0/2104853614 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd844032440 con 0x7fd848101db0 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.285+0000 7fd83f7fe640 1 -- 172.21.15.132:0/2104853614 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd844032a30 con 0x7fd848101db0 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.285+0000 7fd83f7fe640 1 -- 172.21.15.132:0/2104853614 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd844031ad0 con 0x7fd848101db0 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 -- 172.21.15.132:0/2104853614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 msgr2=0x7fd8481021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 --2- 172.21.15.132:0/2104853614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd8481021c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd844009840 tx=0x7fd8440091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 -- 172.21.15.132:0/2104853614 shutdown_connections 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 --2- 172.21.15.132:0/2104853614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd8481021c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 -- 172.21.15.132:0/2104853614 >> 172.21.15.132:0/2104853614 conn(0x7fd8480fd5b0 msgr2=0x7fd8480ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 -- 172.21.15.132:0/2104853614 shutdown_connections 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 -- 172.21.15.132:0/2104853614 wait complete. 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 Processor -- start 2024-06-13T01:18:30.865 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.286+0000 7fd84f00e640 1 -- start start 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84f00e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd848110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84f00e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848110fd0 con 0x7fd848101db0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84cdaa640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd848110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84cdaa640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd848110a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51432/0 (socket says 172.21.15.132:51432) 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84cdaa640 1 -- 172.21.15.132:0/1291444260 learned_addr learned my addr 172.21.15.132:0/1291444260 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84cdaa640 1 -- 172.21.15.132:0/1291444260 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd844009450 con 0x7fd848101db0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84cdaa640 1 --2- 172.21.15.132:0/1291444260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd848110a90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd844009420 tx=0x7fd844008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84403d5b0 con 0x7fd848101db0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8481111d0 con 0x7fd848101db0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd84403dba0 con 0x7fd848101db0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd84403bb40 con 0x7fd848101db0 2024-06-13T01:18:30.866 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.287+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd848111640 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.288+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd844039070 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.288+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd844046950 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.288+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84804b480 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.290+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fd848111640 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.350+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd848111d60 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.350+0000 7fd83d7fa640 1 -- 172.21.15.132:0/1291444260 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fd844041020 con 0x7fd848101db0 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 msgr2=0x7fd848110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 --2- 172.21.15.132:0/1291444260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd848110a90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd844009420 tx=0x7fd844008040 comp rx=0 tx=0).stop 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 shutdown_connections 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 --2- 172.21.15.132:0/1291444260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd848101db0 0x7fd848110a90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 >> 172.21.15.132:0/1291444260 conn(0x7fd8480fd5b0 msgr2=0x7fd8480fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:30.867 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 shutdown_connections 2024-06-13T01:18:30.868 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:29.351+0000 7fd84f00e640 1 -- 172.21.15.132:0/1291444260 wait complete. 2024-06-13T01:18:30.868 INFO:teuthology.orchestra.run.smithi132.stdout:Restarting the monitor... 2024-06-13T01:18:31.127 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:30 smithi132 systemd[1]: Stopping Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:18:31.459 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:31 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132[28956]: 2024-06-13T01:18:31.128+0000 7fef93b50640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi132 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:18:31.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:31 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132[28956]: 2024-06-13T01:18:31.128+0000 7fef93b50640 -1 mon.smithi132@0(leader) e1 *** Got Signal Terminated *** 2024-06-13T01:18:31.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:31 smithi132 podman[29206]: 2024-06-13 01:18:31.354115424 +0000 UTC m=+0.322705353 container died 70c863376bb5af658716a4344e6930905e491c602f300ff8b6431f76df9315ff (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, RELEASE=quincy-296b94f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T01:18:31.959 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:31 smithi132 podman[29206]: 2024-06-13 01:18:31.745539427 +0000 UTC m=+0.714129348 container cleanup 70c863376bb5af658716a4344e6930905e491c602f300ff8b6431f76df9315ff (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_CLEAN=True, RELEASE=quincy-296b94f, org.label-schema.schema-version=1.0) 2024-06-13T01:18:31.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:31 smithi132 bash[29206]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132 2024-06-13T01:18:32.298 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:31 smithi132 podman[29220]: 2024-06-13 01:18:31.963063039 +0000 UTC m=+0.604669354 container remove 70c863376bb5af658716a4344e6930905e491c602f300ff8b6431f76df9315ff (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-296b94f, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-06-13T01:18:32.298 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 systemd[1]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service: Deactivated successfully. 2024-06-13T01:18:32.298 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 systemd[1]: Stopped Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968. 2024-06-13T01:18:32.299 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 systemd[1]: Starting Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 podman[29311]: 2024-06-13 01:18:32.203548208 +0000 UTC m=+0.017079040 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph:quincy 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 podman[29311]: 2024-06-13 01:18:32.305912451 +0000 UTC m=+0.119443256 container create e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, RELEASE=quincy-296b94f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 podman[29311]: 2024-06-13 01:18:32.516580499 +0000 UTC m=+0.330111304 container init e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, ceph=True) 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 podman[29311]: 2024-06-13 01:18:32.51962785 +0000 UTC m=+0.333158655 container start e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-296b94f) 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: set uid:gid to 167:167 (ceph:ceph) 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: ceph version 17.2.7-1006-g296b94f3 (296b94f31805b13f169a9221a78e5c5d2902e4d0) quincy (stable), process ceph-mon, pid 2 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: pidfile_write: ignore empty --pid-file 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: load: jerasure load: lrc 2024-06-13T01:18:32.598 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: RocksDB version: 6.15.5 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Compile date Jun 12 2024 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: DB SUMMARY 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: DB Session ID: 0N0Q0URIQS4IBIQAV4Z9 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: CURRENT file: CURRENT 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: IDENTITY file: IDENTITY 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi132/store.db dir, Total Num: 1, files: 000008.sst 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi132/store.db: 000010.log size: 98198 ; 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.error_if_exists: 0 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.create_if_missing: 0 2024-06-13T01:18:32.599 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.paranoid_checks: 1 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.env: 0x5652d96dc440 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.fs: Posix File System 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.info_log: 0x5652dab1d140 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_file_opening_threads: 16 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.statistics: (nil) 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.use_fsync: 0 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_log_file_size: 0 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.keep_log_file_num: 1000 2024-06-13T01:18:32.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.recycle_log_file_num: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.allow_fallocate: 1 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.allow_mmap_reads: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.allow_mmap_writes: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.use_direct_reads: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.create_missing_column_families: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.db_log_dir: 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi132/store.db 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-13T01:18:32.601 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.advise_random_on_open: 1 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.db_write_buffer_size: 0 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.write_buffer_manager: 0x5652dabda7e0 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.rate_limiter: (nil) 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-13T01:18:32.602 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.wal_recovery_mode: 2 2024-06-13T01:18:32.603 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.enable_thread_tracking: 0 2024-06-13T01:18:32.603 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.enable_pipelined_write: 0 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.unordered_write: 0 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.row_cache: None 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.wal_filter: None 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-13T01:18:32.604 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.allow_ingest_behind: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.preserve_deletes: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.two_write_queues: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.manual_wal_flush: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.atomic_flush: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.log_readahead_size: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.best_efforts_recovery: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.allow_data_in_errors: 0 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.db_host_id: __hostname__ 2024-06-13T01:18:32.605 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_background_jobs: 2 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_background_compactions: -1 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_subcompactions: 1 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_total_wal_size: 0 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_open_files: -1 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bytes_per_sync: 0 2024-06-13T01:18:32.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_readahead_size: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_background_flushes: -1 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Compression algorithms supported: 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kZSTD supported: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kXpressCompression supported: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kBZip2Compression supported: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kLZ4Compression supported: 1 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kZlibCompression supported: 1 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kLZ4HCCompression supported: 1 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: kSnappyCompression supported: 1 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi132/store.db/MANIFEST-000009 2024-06-13T01:18:32.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.merge_operator: 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_filter: None 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_filter_factory: None 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.sst_partitioner_factory: None 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5652daac3a68) 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: cache_index_and_filter_blocks: 1 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: pin_top_level_index_and_filter: 1 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: index_type: 0 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: data_block_index_type: 0 2024-06-13T01:18:32.608 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: index_shortening: 1 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: hash_index_allow_collision: 1 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: checksum: 1 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: no_block_cache: 0 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_cache: 0x5652daaf91f0 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_cache_name: BinnedLRUCache 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_cache_options: 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: capacity : 536870912 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: num_shard_bits : 4 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: strict_capacity_limit : 0 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: high_pri_pool_ratio: 0.000 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_cache_compressed: (nil) 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: persistent_cache: (nil) 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_size: 4096 2024-06-13T01:18:32.609 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_size_deviation: 10 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_restart_interval: 16 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: index_block_restart_interval: 1 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: metadata_block_size: 4096 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: partition_filters: 0 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: use_delta_encoding: 1 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: whole_key_filtering: 1 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: verify_compression: 0 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: read_amp_bytes_per_bit: 0 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: format_version: 4 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: enable_index_compression: 1 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: block_align: 0 2024-06-13T01:18:32.610 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.write_buffer_size: 33554432 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_write_buffer_number: 2 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression: NoCompression 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression: Disabled 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.prefix_extractor: nullptr 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.num_levels: 7 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-13T01:18:32.611 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.level: 32767 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.strategy: 0 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compression_opts.enabled: false 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-13T01:18:32.612 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.target_file_size_base: 67108864 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-13T01:18:32.613 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-13T01:18:32.614 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-13T01:18:32.614 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-13T01:18:32.614 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-13T01:18:32.614 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-13T01:18:32.614 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.arena_block_size: 4194304 2024-06-13T01:18:32.615 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-13T01:18:32.615 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-13T01:18:32.615 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-13T01:18:32.615 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.disable_auto_compactions: 0 2024-06-13T01:18:32.615 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-13T01:18:32.615 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-13T01:18:32.616 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.table_properties_collectors: 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.inplace_update_support: 0 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-13T01:18:32.617 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.bloom_locality: 0 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.max_successive_merges: 0 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.paranoid_file_checks: 0 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.force_consistency_checks: 1 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.report_bg_io_stats: 0 2024-06-13T01:18:32.618 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.ttl: 2592000 2024-06-13T01:18:32.619 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-13T01:18:32.619 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.enable_blob_files: false 2024-06-13T01:18:32.619 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.min_blob_size: 0 2024-06-13T01:18:32.619 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.blob_file_size: 268435456 2024-06-13T01:18:32.619 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-13T01:18:32.619 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-13T01:18:32.620 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-13T01:18:32.620 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi132/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-13T01:18:32.620 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-13T01:18:32.620 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-13T01:18:32.620 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718241512543498, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-13T01:18:32.620 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718241512544668, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93404, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91516, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84080, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718241512, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "e4635ef8-c497-43ea-b492-1fd87c61cd1c", "db_session_id": "0N0Q0URIQS4IBIQAV4Z9"}} 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718241512545119, "job": 1, "event": "recovery_finished"} 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi132/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5652dab2c700 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: DB pointer 0x5652daafc000 2024-06-13T01:18:32.621 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** DB Stats ** 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.622 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** Compaction Stats [default] ** 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: L0 2/0 93.01 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Sum 2/0 93.01 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** Compaction Stats [default] ** 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-13T01:18:32.623 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Cumulative compaction: 0.00 GB write, 20.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Interval compaction: 0.00 GB write, 20.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** Compaction Stats [default] ** 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-13T01:18:32.624 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: L0 2/0 93.01 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Sum 2/0 93.01 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** Compaction Stats [default] ** 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.9 0.00 0.00 1 0.001 0 0 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-13T01:18:32.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Cumulative compaction: 0.00 GB write, 20.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: starting mon.smithi132 rank 0 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi132 fsid ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???) e1 preinit fsid ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).mds e1 new map 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).mds e1 print_map 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: e1 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-13T01:18:32.626 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: legacy client fscid: -1 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: No filesystems configured 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-13T01:18:32.627 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 bash[29311]: e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef 2024-06-13T01:18:32.627 INFO:teuthology.orchestra.run.smithi132.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-13T01:18:32.933 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 systemd[1]: Started Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968. 2024-06-13T01:18:32.933 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mon.smithi132 is new leader, mons smithi132 in quorum (ranks 0) 2024-06-13T01:18:32.933 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: monmap e1: 1 mons at {smithi132=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-13T01:18:32.933 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: fsmap 2024-06-13T01:18:32.933 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: osdmap e1: 0 total, 0 up, 0 in 2024-06-13T01:18:32.934 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:32 smithi132 ceph-mon[29330]: mgrmap e1: no daemons active 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.013+0000 7f1943331640 1 Processor -- start 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.013+0000 7f1943331640 1 -- start start 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.014+0000 7f1943331640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.014+0000 7f1943331640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1015a0 con 0x7f193c100c50 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.014+0000 7f19410cd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.014+0000 7f19410cd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46018/0 (socket says 172.21.15.132:46018) 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.014+0000 7f19410cd640 1 -- 172.21.15.132:0/4280514725 learned_addr learned my addr 172.21.15.132:0/4280514725 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:33.636 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.014+0000 7f19410cd640 1 -- 172.21.15.132:0/4280514725 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c1016e0 con 0x7f193c100c50 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.016+0000 7f19410cd640 1 --2- 172.21.15.132:0/4280514725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c101060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f192c009e50 tx=0x7f192c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af29cf7d0745b1ff server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.016+0000 7f193bfff640 1 -- 172.21.15.132:0/4280514725 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192c031560 con 0x7f193c100c50 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.016+0000 7f193bfff640 1 -- 172.21.15.132:0/4280514725 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f192c02d050 con 0x7f193c100c50 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.016+0000 7f193bfff640 1 -- 172.21.15.132:0/4280514725 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192c007a50 con 0x7f193c100c50 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 -- 172.21.15.132:0/4280514725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 msgr2=0x7f193c101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 --2- 172.21.15.132:0/4280514725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c101060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f192c009e50 tx=0x7f192c009840 comp rx=0 tx=0).stop 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 -- 172.21.15.132:0/4280514725 shutdown_connections 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 --2- 172.21.15.132:0/4280514725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c101060 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 -- 172.21.15.132:0/4280514725 >> 172.21.15.132:0/4280514725 conn(0x7f193c0fc430 msgr2=0x7f193c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 -- 172.21.15.132:0/4280514725 shutdown_connections 2024-06-13T01:18:33.637 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 -- 172.21.15.132:0/4280514725 wait complete. 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 Processor -- start 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.017+0000 7f1943331640 1 -- start start 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f1943331640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c10e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f1943331640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c10eea0 con 0x7f193c100c50 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f19410cd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c10e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f19410cd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c10e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46020/0 (socket says 172.21.15.132:46020) 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f19410cd640 1 -- 172.21.15.132:0/1931374801 learned_addr learned my addr 172.21.15.132:0/1931374801 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f19410cd640 1 -- 172.21.15.132:0/1931374801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c009ae0 con 0x7f193c100c50 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f19410cd640 1 --2- 172.21.15.132:0/1931374801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c10e960 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f192c0319b0 tx=0x7f192c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192c031df0 con 0x7f193c100c50 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c10f0a0 con 0x7f193c100c50 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f192c0458c0 con 0x7f193c100c50 2024-06-13T01:18:33.638 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.018+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192c03a8c0 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.019+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c10f510 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.019+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f192c038040 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.019+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f192c040030 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.020+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c04b480 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.022+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f193c10f510 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.079+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f193c1029c0 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.084+0000 7f1939ffb640 1 -- 172.21.15.132:0/1931374801 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f192c036070 con 0x7f193c100c50 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 msgr2=0x7f193c10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 --2- 172.21.15.132:0/1931374801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c10e960 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f192c0319b0 tx=0x7f192c009840 comp rx=0 tx=0).stop 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 shutdown_connections 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 --2- 172.21.15.132:0/1931374801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f193c100c50 0x7f193c10e960 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 >> 172.21.15.132:0/1931374801 conn(0x7f193c0fc430 msgr2=0x7f193c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 shutdown_connections 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:33.085+0000 7f1943331640 1 -- 172.21.15.132:0/1931374801 wait complete. 2024-06-13T01:18:33.639 INFO:teuthology.orchestra.run.smithi132.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-13T01:18:33.640 INFO:teuthology.orchestra.run.smithi132.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:18:33.640 INFO:teuthology.orchestra.run.smithi132.stdout:Creating mgr... 2024-06-13T01:18:33.640 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying port 9283 ... 2024-06-13T01:18:33.812 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mgr.smithi132.vjweoa 2024-06-13T01:18:33.813 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mgr.smithi132.vjweoa.service: Unit ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mgr.smithi132.vjweoa.service not loaded. 2024-06-13T01:18:33.958 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968.target.wants/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mgr.smithi132.vjweoa.service → /etc/systemd/system/ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@.service. 2024-06-13T01:18:34.365 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:34 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1931374801' entity='client.admin' 2024-06-13T01:18:34.489 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T01:18:34.489 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout disabled 2024-06-13T01:18:34.497 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T01:18:34.497 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-06-13T01:18:34.497 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld.service is not enabled 2024-06-13T01:18:34.497 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to enable service . firewalld.service is not available 2024-06-13T01:18:34.506 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T01:18:34.507 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout disabled 2024-06-13T01:18:34.515 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T01:18:34.516 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-06-13T01:18:34.516 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld.service is not enabled 2024-06-13T01:18:34.516 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-13T01:18:34.516 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr to start... 2024-06-13T01:18:34.516 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr... 2024-06-13T01:18:35.209 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:35 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/4130689924' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-13T01:18:35.592 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "ad8d4854-2922-11ef-bc9c-c7b262605968", 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-13T01:18:35.593 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "smithi132" 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-13T01:18:35.594 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.595 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T01:18:35.596 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-06-13T01:18:26.792055+0000", 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.922+0000 7f3bb184e640 1 Processor -- start 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.922+0000 7f3bb184e640 1 -- start start 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.923+0000 7f3bb184e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac076220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.923+0000 7f3bb184e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0780b0 con 0x7f3bac077ca0 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.923+0000 7f3baaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.923+0000 7f3baaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46044/0 (socket says 172.21.15.132:46044) 2024-06-13T01:18:35.597 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.923+0000 7f3baaffd640 1 -- 172.21.15.132:0/85248344 learned_addr learned my addr 172.21.15.132:0/85248344 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.923+0000 7f3baaffd640 1 -- 172.21.15.132:0/85248344 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac076760 con 0x7f3bac077ca0 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.924+0000 7f3baaffd640 1 --2- 172.21.15.132:0/85248344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac076220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009840 tx=0x7f3ba00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8691c5c8fe5b5cae server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.924+0000 7f3ba9ffb640 1 -- 172.21.15.132:0/85248344 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba0032440 con 0x7f3bac077ca0 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.924+0000 7f3ba9ffb640 1 -- 172.21.15.132:0/85248344 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3ba0032a30 con 0x7f3bac077ca0 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.924+0000 7f3ba9ffb640 1 -- 172.21.15.132:0/85248344 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba0031ad0 con 0x7f3bac077ca0 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 -- 172.21.15.132:0/85248344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 msgr2=0x7f3bac076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 --2- 172.21.15.132:0/85248344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac076220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009840 tx=0x7f3ba00091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 -- 172.21.15.132:0/85248344 shutdown_connections 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 --2- 172.21.15.132:0/85248344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac076220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 -- 172.21.15.132:0/85248344 >> 172.21.15.132:0/85248344 conn(0x7f3bac0fd490 msgr2=0x7f3bac0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 -- 172.21.15.132:0/85248344 shutdown_connections 2024-06-13T01:18:35.598 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.925+0000 7f3bb184e640 1 -- 172.21.15.132:0/85248344 wait complete. 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3bb184e640 1 Processor -- start 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3bb184e640 1 -- start start 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3bb184e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac112d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3bb184e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac113250 con 0x7f3bac077ca0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3baaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac112d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3baaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac112d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46046/0 (socket says 172.21.15.132:46046) 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3baaffd640 1 -- 172.21.15.132:0/4130689924 learned_addr learned my addr 172.21.15.132:0/4130689924 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3baaffd640 1 -- 172.21.15.132:0/4130689924 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba0009450 con 0x7f3bac077ca0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.926+0000 7f3baaffd640 1 --2- 172.21.15.132:0/4130689924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac112d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0031b90 tx=0x7f3ba0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba00328b0 con 0x7f3bac077ca0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3ba003d8e0 con 0x7f3bac077ca0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba003ba20 con 0x7f3bac077ca0 2024-06-13T01:18:35.599 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac113450 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac1138c0 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f3ba0039070 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.927+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3ba00468e0 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.928+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bac04b480 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.930+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f3ba0065790 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.993+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3bac1140a0 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.994+0000 7f3b937fe640 1 -- 172.21.15.132:0/4130689924 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f3ba0054020 con 0x7f3bac077ca0 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 msgr2=0x7f3bac112d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 --2- 172.21.15.132:0/4130689924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac112d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0031b90 tx=0x7f3ba0008040 comp rx=0 tx=0).stop 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 shutdown_connections 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 --2- 172.21.15.132:0/4130689924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3bac077ca0 0x7f3bac112d10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 >> 172.21.15.132:0/4130689924 conn(0x7f3bac0fd490 msgr2=0x7f3bac0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:35.600 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 shutdown_connections 2024-06-13T01:18:35.601 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:34.995+0000 7f3bb184e640 1 -- 172.21.15.132:0/4130689924 wait complete. 2024-06-13T01:18:35.601 INFO:teuthology.orchestra.run.smithi132.stdout:mgr not available, waiting (1/15)... 2024-06-13T01:18:38.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:38 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/4068712189' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "ad8d4854-2922-11ef-bc9c-c7b262605968", 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "smithi132" 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-13T01:18:38.802 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-13T01:18:38.803 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-06-13T01:18:26.792055+0000", 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:38.804 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.101+0000 7ff2f850d640 1 Processor -- start 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.101+0000 7ff2f850d640 1 -- start start 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.102+0000 7ff2f850d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f01021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.102+0000 7ff2f850d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0102700 con 0x7ff2f0101db0 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.102+0000 7ff2f62a9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f01021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.102+0000 7ff2f62a9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f01021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46060/0 (socket says 172.21.15.132:46060) 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.102+0000 7ff2f62a9640 1 -- 172.21.15.132:0/49617477 learned_addr learned my addr 172.21.15.132:0/49617477 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.102+0000 7ff2f62a9640 1 -- 172.21.15.132:0/49617477 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0102840 con 0x7ff2f0101db0 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.103+0000 7ff2f62a9640 1 --2- 172.21.15.132:0/49617477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f01021c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009840 tx=0x7ff2ec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4705ebef0f72cf89 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.103+0000 7ff2f52a7640 1 -- 172.21.15.132:0/49617477 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ec032440 con 0x7ff2f0101db0 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.103+0000 7ff2f52a7640 1 -- 172.21.15.132:0/49617477 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff2ec032a30 con 0x7ff2f0101db0 2024-06-13T01:18:38.805 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.103+0000 7ff2f52a7640 1 -- 172.21.15.132:0/49617477 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ec031ad0 con 0x7ff2f0101db0 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 -- 172.21.15.132:0/49617477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 msgr2=0x7ff2f01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 --2- 172.21.15.132:0/49617477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f01021c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009840 tx=0x7ff2ec0091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 -- 172.21.15.132:0/49617477 shutdown_connections 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 --2- 172.21.15.132:0/49617477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f01021c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 -- 172.21.15.132:0/49617477 >> 172.21.15.132:0/49617477 conn(0x7ff2f00fd5b0 msgr2=0x7ff2f00ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 -- 172.21.15.132:0/49617477 shutdown_connections 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.104+0000 7ff2f850d640 1 -- 172.21.15.132:0/49617477 wait complete. 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f850d640 1 Processor -- start 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f850d640 1 -- start start 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f850d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f0110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f850d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0110fd0 con 0x7ff2f0101db0 2024-06-13T01:18:38.806 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f62a9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f0110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f62a9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f0110a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46072/0 (socket says 172.21.15.132:46072) 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f62a9640 1 -- 172.21.15.132:0/4068712189 learned_addr learned my addr 172.21.15.132:0/4068712189 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f62a9640 1 -- 172.21.15.132:0/4068712189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec009450 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.105+0000 7ff2f62a9640 1 --2- 172.21.15.132:0/4068712189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f0110a90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0317b0 tx=0x7ff2ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ec031e90 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff2ec03daf0 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f01111d0 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ec03baa0 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f0111640 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ff2ec039070 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.106+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff2ec046990 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.107+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f004b480 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.109+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7ff2f0111640 con 0x7ff2f0101db0 2024-06-13T01:18:38.807 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.171+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff2f0111d60 con 0x7ff2f0101db0 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.172+0000 7ff2e6ffd640 1 -- 172.21.15.132:0/4068712189 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7ff2ec041020 con 0x7ff2f0101db0 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 msgr2=0x7ff2f0110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 --2- 172.21.15.132:0/4068712189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f0110a90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0317b0 tx=0x7ff2ec008040 comp rx=0 tx=0).stop 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 shutdown_connections 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 --2- 172.21.15.132:0/4068712189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2f0101db0 0x7ff2f0110a90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 >> 172.21.15.132:0/4068712189 conn(0x7ff2f00fd5b0 msgr2=0x7ff2f00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 shutdown_connections 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:38.173+0000 7ff2f850d640 1 -- 172.21.15.132:0/4068712189 wait complete. 2024-06-13T01:18:38.808 INFO:teuthology.orchestra.run.smithi132.stdout:mgr not available, waiting (2/15)... 2024-06-13T01:18:39.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: Activating manager daemon smithi132.vjweoa 2024-06-13T01:18:39.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: mgrmap e2: smithi132.vjweoa(active, starting, since 0.00299913s) 2024-06-13T01:18:39.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T01:18:39.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T01:18:39.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr metadata", "who": "smithi132.vjweoa", "id": "smithi132.vjweoa"}]: dispatch 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: Manager daemon smithi132.vjweoa is now available 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/mirror_snapshot_schedule"}]: dispatch 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/trash_purge_schedule"}]: dispatch 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:39.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:39 smithi132 ceph-mon[29330]: from='mgr.14100 172.21.15.132:0/3578364301' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:41.205 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:40 smithi132 ceph-mon[29330]: mgrmap e3: smithi132.vjweoa(active, since 1.00696s) 2024-06-13T01:18:42.032 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:42.032 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "ad8d4854-2922-11ef-bc9c-c7b262605968", 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "smithi132" 2024-06-13T01:18:42.033 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-13T01:18:42.034 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-13T01:18:42.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-13T01:18:42.043 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-06-13T01:18:26.792055+0000", 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-06-13T01:18:42.044 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.207+0000 7f6394ca2640 1 Processor -- start 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.207+0000 7f6394ca2640 1 -- start start 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.207+0000 7f6394ca2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f63901020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.207+0000 7f6394ca2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390102600 con 0x7f6390101cb0 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.208+0000 7f638e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f63901020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.208+0000 7f638e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f63901020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46140/0 (socket says 172.21.15.132:46140) 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.208+0000 7f638e59c640 1 -- 172.21.15.132:0/1293695452 learned_addr learned my addr 172.21.15.132:0/1293695452 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.208+0000 7f638e59c640 1 -- 172.21.15.132:0/1293695452 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6390102740 con 0x7f6390101cb0 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f638e59c640 1 --2- 172.21.15.132:0/1293695452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f63901020c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6380009840 tx=0x7f63800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dfa839fbbc60130 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f638dd9b640 1 -- 172.21.15.132:0/1293695452 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6380032440 con 0x7f6390101cb0 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f638dd9b640 1 -- 172.21.15.132:0/1293695452 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6380032a30 con 0x7f6390101cb0 2024-06-13T01:18:42.045 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f638dd9b640 1 -- 172.21.15.132:0/1293695452 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6380031ad0 con 0x7f6390101cb0 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f6394ca2640 1 -- 172.21.15.132:0/1293695452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 msgr2=0x7f63901020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f6394ca2640 1 --2- 172.21.15.132:0/1293695452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f63901020c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6380009840 tx=0x7f63800091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f6394ca2640 1 -- 172.21.15.132:0/1293695452 shutdown_connections 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f6394ca2640 1 --2- 172.21.15.132:0/1293695452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f63901020c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.209+0000 7f6394ca2640 1 -- 172.21.15.132:0/1293695452 >> 172.21.15.132:0/1293695452 conn(0x7f63900fd490 msgr2=0x7f63900ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.210+0000 7f6394ca2640 1 -- 172.21.15.132:0/1293695452 shutdown_connections 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.210+0000 7f6394ca2640 1 -- 172.21.15.132:0/1293695452 wait complete. 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.210+0000 7f6394ca2640 1 Processor -- start 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.210+0000 7f6394ca2640 1 -- start start 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.210+0000 7f6394ca2640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f6390110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.210+0000 7f6394ca2640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63901113e0 con 0x7f6390101cb0 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f638e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f6390110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:42.046 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f638e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f6390110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46154/0 (socket says 172.21.15.132:46154) 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f638e59c640 1 -- 172.21.15.132:0/846730046 learned_addr learned my addr 172.21.15.132:0/846730046 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f638e59c640 1 -- 172.21.15.132:0/846730046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6380009450 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f638e59c640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f6390110ea0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63800317b0 tx=0x7f6380008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6380031e90 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f638003daf0 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638003baa0 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63901115e0 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.211+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6390111fe0 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.212+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f6380039070 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.212+0000 7f63877fe640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f637c034810 0x7f637c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.212+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f63800674c0 con 0x7f6390101cb0 2024-06-13T01:18:42.047 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.212+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639005ead0 con 0x7f6390101cb0 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.212+0000 7f6385d9b640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f637c034810 0x7f637c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.213+0000 7f6385d9b640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f637c034810 0x7f637c036cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6378006fd0 tx=0x7f6378006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.217+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6380041080 con 0x7f6390101cb0 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.442+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6390063ed0 con 0x7f6390101cb0 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.443+0000 7f63877fe640 1 -- 172.21.15.132:0/846730046 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f6380037070 con 0x7f6390101cb0 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f637c034810 msgr2=0x7f637c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f637c034810 0x7f637c036cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6378006fd0 tx=0x7f6378006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 msgr2=0x7f6390110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f6390110ea0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f63800317b0 tx=0x7f6380008040 comp rx=0 tx=0).stop 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 shutdown_connections 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f637c034810 0x7f637c036cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 --2- 172.21.15.132:0/846730046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6390101cb0 0x7f6390110ea0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 >> 172.21.15.132:0/846730046 conn(0x7f63900fd490 msgr2=0x7f63900fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 shutdown_connections 2024-06-13T01:18:42.049 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:41.445+0000 7f6394ca2640 1 -- 172.21.15.132:0/846730046 wait complete. 2024-06-13T01:18:42.049 INFO:teuthology.orchestra.run.smithi132.stdout:mgr is available 2024-06-13T01:18:42.145 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:41 smithi132 ceph-mon[29330]: mgrmap e4: smithi132.vjweoa(active, since 2s) 2024-06-13T01:18:42.145 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:41 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/846730046' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-13T01:18:43.185 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:42 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1206571497' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-13T01:18:43.251 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [global] 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout fsid = ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [osd] 2024-06-13T01:18:43.252 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2e01430640 1 Processor -- start 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2e01430640 1 -- start start 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2e01430640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc1020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2e01430640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc102600 con 0x7f2dfc101cb0 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2dfaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc1020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2dfaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc1020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36048/0 (socket says 172.21.15.132:36048) 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2dfaffd640 1 -- 172.21.15.132:0/1169628375 learned_addr learned my addr 172.21.15.132:0/1169628375 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.410+0000 7f2dfaffd640 1 -- 172.21.15.132:0/1169628375 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc102740 con 0x7f2dfc101cb0 2024-06-13T01:18:43.253 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.411+0000 7f2dfaffd640 1 --2- 172.21.15.132:0/1169628375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc1020c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009840 tx=0x7f2dec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c219fa91698d server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.411+0000 7f2df9ffb640 1 -- 172.21.15.132:0/1169628375 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec032440 con 0x7f2dfc101cb0 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.411+0000 7f2df9ffb640 1 -- 172.21.15.132:0/1169628375 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2dec032a30 con 0x7f2dfc101cb0 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.411+0000 7f2df9ffb640 1 -- 172.21.15.132:0/1169628375 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec031ad0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 -- 172.21.15.132:0/1169628375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 msgr2=0x7f2dfc1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 --2- 172.21.15.132:0/1169628375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc1020c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009840 tx=0x7f2dec0091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 -- 172.21.15.132:0/1169628375 shutdown_connections 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 --2- 172.21.15.132:0/1169628375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc1020c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 -- 172.21.15.132:0/1169628375 >> 172.21.15.132:0/1169628375 conn(0x7f2dfc0fd490 msgr2=0x7f2dfc0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 -- 172.21.15.132:0/1169628375 shutdown_connections 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 -- 172.21.15.132:0/1169628375 wait complete. 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 Processor -- start 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.412+0000 7f2e01430640 1 -- start start 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2e01430640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:43.254 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2e01430640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc1113e0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2dfaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2dfaffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36064/0 (socket says 172.21.15.132:36064) 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2dfaffd640 1 -- 172.21.15.132:0/1206571497 learned_addr learned my addr 172.21.15.132:0/1206571497 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2dfaffd640 1 -- 172.21.15.132:0/1206571497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec009450 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2dfaffd640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc110ea0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0317b0 tx=0x7f2dec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec031e90 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2dec03daf0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc1115e0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec03baa0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.413+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc111fe0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.414+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f2dec039070 con 0x7f2dfc101cb0 2024-06-13T01:18:43.255 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.414+0000 7f2de37fe640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f2de40347c0 0x7f2de4036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.414+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2dec067590 con 0x7f2dfc101cb0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.414+0000 7f2dfa7fc640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f2de40347c0 0x7f2de4036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.414+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc05ead0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.415+0000 7f2dfa7fc640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f2de40347c0 0x7f2de4036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006fd0 tx=0x7f2df0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.419+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2dec041080 con 0x7f2dfc101cb0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.583+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2dfc063ed0 con 0x7f2dfc101cb0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.584+0000 7f2de37fe640 1 -- 172.21.15.132:0/1206571497 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2dec066190 con 0x7f2dfc101cb0 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f2de40347c0 msgr2=0x7f2de4036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f2de40347c0 0x7f2de4036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006fd0 tx=0x7f2df0006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 msgr2=0x7f2dfc110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc110ea0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0317b0 tx=0x7f2dec008040 comp rx=0 tx=0).stop 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 shutdown_connections 2024-06-13T01:18:43.256 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f2de40347c0 0x7f2de4036c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:43.257 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 --2- 172.21.15.132:0/1206571497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dfc101cb0 0x7f2dfc110ea0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:43.257 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 >> 172.21.15.132:0/1206571497 conn(0x7f2dfc0fd490 msgr2=0x7f2dfc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:43.257 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 shutdown_connections 2024-06-13T01:18:43.257 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:42.586+0000 7f2e01430640 1 -- 172.21.15.132:0/1206571497 wait complete. 2024-06-13T01:18:43.257 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling cephadm module... 2024-06-13T01:18:44.167 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:43 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/524988597' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-13T01:18:44.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.608+0000 7f0d05c15640 1 Processor -- start 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.608+0000 7f0d05c15640 1 -- start start 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.608+0000 7f0d05c15640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00106d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.608+0000 7f0d05c15640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d00102770 con 0x7f0d00075b90 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.609+0000 7f0cff7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00106d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.609+0000 7f0cff7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00106d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36080/0 (socket says 172.21.15.132:36080) 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.609+0000 7f0cff7fe640 1 -- 172.21.15.132:0/1734380502 learned_addr learned my addr 172.21.15.132:0/1734380502 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.609+0000 7f0cff7fe640 1 -- 172.21.15.132:0/1734380502 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d001072c0 con 0x7f0d00075b90 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0cff7fe640 1 --2- 172.21.15.132:0/1734380502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00106d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009840 tx=0x7f0cf00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8681bc1844c2fc73 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0cfe7fc640 1 -- 172.21.15.132:0/1734380502 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cf0032440 con 0x7f0d00075b90 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0cfe7fc640 1 -- 172.21.15.132:0/1734380502 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0cf0032a30 con 0x7f0d00075b90 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0cfe7fc640 1 -- 172.21.15.132:0/1734380502 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cf0031ad0 con 0x7f0d00075b90 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 -- 172.21.15.132:0/1734380502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 msgr2=0x7f0d00106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:44.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 --2- 172.21.15.132:0/1734380502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00106d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009840 tx=0x7f0cf00091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 -- 172.21.15.132:0/1734380502 shutdown_connections 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 --2- 172.21.15.132:0/1734380502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00106d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 -- 172.21.15.132:0/1734380502 >> 172.21.15.132:0/1734380502 conn(0x7f0d000fc4b0 msgr2=0x7f0d000fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 -- 172.21.15.132:0/1734380502 shutdown_connections 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.610+0000 7f0d05c15640 1 -- 172.21.15.132:0/1734380502 wait complete. 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0d05c15640 1 Processor -- start 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0d05c15640 1 -- start start 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0d05c15640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00112cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0d05c15640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d00113200 con 0x7f0d00075b90 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0cff7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00112cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0cff7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00112cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36084/0 (socket says 172.21.15.132:36084) 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.611+0000 7f0cff7fe640 1 -- 172.21.15.132:0/524988597 learned_addr learned my addr 172.21.15.132:0/524988597 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:44.517 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0cff7fe640 1 -- 172.21.15.132:0/524988597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0009450 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0cff7fe640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00112cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0031b90 tx=0x7f0cf0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cf00328b0 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0cf003d8e0 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d00113400 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cf003ba20 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.612+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d00113870 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.613+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f0cf0039070 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.613+0000 7f0ce7fff640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f0ce8034810 0x7f0ce8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.613+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0cf00674d0 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.613+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0005ead0 con 0x7f0d00075b90 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.613+0000 7f0cfeffd640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f0ce8034810 0x7f0ce8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.613+0000 7f0cfeffd640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f0ce8034810 0x7f0ce8036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4006fd0 tx=0x7f0cf4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:44.518 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.617+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0cf0041020 con 0x7f0d00075b90 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.823+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f0d00114090 con 0x7f0d00075b90 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.910+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0cf00464f0 con 0x7f0d00075b90 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.911+0000 7f0ce7fff640 1 -- 172.21.15.132:0/524988597 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f0cf0067230 con 0x7f0d00075b90 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f0ce8034810 msgr2=0x7f0ce8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f0ce8034810 0x7f0ce8036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4006fd0 tx=0x7f0cf4006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 msgr2=0x7f0d00112cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00112cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0031b90 tx=0x7f0cf0008040 comp rx=0 tx=0).stop 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 shutdown_connections 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f0ce8034810 0x7f0ce8036cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 --2- 172.21.15.132:0/524988597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d00075b90 0x7f0d00112cc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.913+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 >> 172.21.15.132:0/524988597 conn(0x7f0d000fc4b0 msgr2=0x7f0d000fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.914+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 shutdown_connections 2024-06-13T01:18:44.519 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:43.914+0000 7f0d05c15640 1 -- 172.21.15.132:0/524988597 wait complete. 2024-06-13T01:18:45.119 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:44 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/524988597' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-13T01:18:45.119 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:44 smithi132 ceph-mon[29330]: mgrmap e5: smithi132.vjweoa(active, since 5s) 2024-06-13T01:18:45.773 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:18:45.773 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-13T01:18:45.773 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-13T01:18:45.773 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "active_name": "smithi132.vjweoa", 2024-06-13T01:18:45.773 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.914+0000 7f6f58315640 1 Processor -- start 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.914+0000 7f6f58315640 1 -- start start 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.915+0000 7f6f58315640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f501049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.915+0000 7f6f58315640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50075f20 con 0x7f6f50075b10 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.915+0000 7f6f560b1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f501049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.915+0000 7f6f560b1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f501049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36112/0 (socket says 172.21.15.132:36112) 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.915+0000 7f6f560b1640 1 -- 172.21.15.132:0/963778893 learned_addr learned my addr 172.21.15.132:0/963778893 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.915+0000 7f6f560b1640 1 -- 172.21.15.132:0/963778893 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50104f20 con 0x7f6f50075b10 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.916+0000 7f6f560b1640 1 --2- 172.21.15.132:0/963778893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f501049e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009840 tx=0x7f6f4c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a225efde48b8a3af server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.916+0000 7f6f550af640 1 -- 172.21.15.132:0/963778893 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4c032440 con 0x7f6f50075b10 2024-06-13T01:18:45.774 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.916+0000 7f6f550af640 1 -- 172.21.15.132:0/963778893 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6f4c032a30 con 0x7f6f50075b10 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.916+0000 7f6f550af640 1 -- 172.21.15.132:0/963778893 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4c031ad0 con 0x7f6f50075b10 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- 172.21.15.132:0/963778893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 msgr2=0x7f6f501049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 --2- 172.21.15.132:0/963778893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f501049e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009840 tx=0x7f6f4c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- 172.21.15.132:0/963778893 shutdown_connections 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 --2- 172.21.15.132:0/963778893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f501049e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- 172.21.15.132:0/963778893 >> 172.21.15.132:0/963778893 conn(0x7f6f500fc490 msgr2=0x7f6f500fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- 172.21.15.132:0/963778893 shutdown_connections 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- 172.21.15.132:0/963778893 wait complete. 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 Processor -- start 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- start start 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f50110ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:45.775 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f58315640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50111000 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.917+0000 7f6f560b1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f50110ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f560b1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f50110ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36122/0 (socket says 172.21.15.132:36122) 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f560b1640 1 -- 172.21.15.132:0/460489509 learned_addr learned my addr 172.21.15.132:0/460489509 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f560b1640 1 -- 172.21.15.132:0/460489509 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c009450 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f560b1640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f50110ac0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c031b90 tx=0x7f6f4c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4c0328b0 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6f4c03d8e0 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4c03ba20 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f50111200 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.918+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f50111670 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.919+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5004b480 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.919+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f6f4c039070 con 0x7f6f50075b10 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.919+0000 7f6f46ffd640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 0x7f6f3c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.920+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6f4c0672d0 con 0x7f6f50075b10 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.920+0000 7f6f558b0640 1 -- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 msgr2=0x7f6f3c036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1802099589 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.920+0000 7f6f558b0640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 0x7f6f3c036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:44.924+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6f4c03ca00 con 0x7f6f50075b10 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.121+0000 7f6f558b0640 1 -- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 msgr2=0x7f6f3c036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1802099589 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.121+0000 7f6f558b0640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 0x7f6f3c036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.140+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f6f50105cd0 con 0x7f6f50075b10 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.140+0000 7f6f46ffd640 1 -- 172.21.15.132:0/460489509 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f6f4c041070 con 0x7f6f50075b10 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 msgr2=0x7f6f3c036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 0x7f6f3c036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 msgr2=0x7f6f50110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f50110ac0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c031b90 tx=0x7f6f4c008040 comp rx=0 tx=0).stop 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 shutdown_connections 2024-06-13T01:18:45.777 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7f6f3c034860 0x7f6f3c036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:45.778 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 --2- 172.21.15.132:0/460489509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50075b10 0x7f6f50110ac0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:45.778 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.142+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 >> 172.21.15.132:0/460489509 conn(0x7f6f500fc490 msgr2=0x7f6f500fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:45.778 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.143+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 shutdown_connections 2024-06-13T01:18:45.778 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:45.143+0000 7f6f58315640 1 -- 172.21.15.132:0/460489509 wait complete. 2024-06-13T01:18:45.778 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for the mgr to restart... 2024-06-13T01:18:45.778 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr epoch 5... 2024-06-13T01:18:46.017 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:45 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/460489509' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: Active manager daemon smithi132.vjweoa restarted 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: Activating manager daemon smithi132.vjweoa 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: osdmap e2: 0 total, 0 up, 0 in 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: mgrmap e6: smithi132.vjweoa(active, starting, since 0.00328173s) 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr metadata", "who": "smithi132.vjweoa", "id": "smithi132.vjweoa"}]: dispatch 2024-06-13T01:18:48.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: Manager daemon smithi132.vjweoa is now available 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/mirror_snapshot_schedule"}]: dispatch 2024-06-13T01:18:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:48 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/trash_purge_schedule"}]: dispatch 2024-06-13T01:18:49.705 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:49 smithi132 ceph-mon[29330]: Found migration_current of "None". Setting to last migration. 2024-06-13T01:18:49.705 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:49 smithi132 ceph-mon[29330]: mgrmap e7: smithi132.vjweoa(active, since 1.00493s) 2024-06-13T01:18:49.941 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:18:49.941 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-13T01:18:49.941 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-13T01:18:49.941 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:18:49.941 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86fd50640 1 Processor -- start 2024-06-13T01:18:49.941 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86fd50640 1 -- start start 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86fd50640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86fd50640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff868075f50 con 0x7ff868075b40 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86daec640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86daec640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36136/0 (socket says 172.21.15.132:36136) 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.195+0000 7ff86daec640 1 -- 172.21.15.132:0/2032087853 learned_addr learned my addr 172.21.15.132:0/2032087853 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.196+0000 7ff86daec640 1 -- 172.21.15.132:0/2032087853 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff868104ee0 con 0x7ff868075b40 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.196+0000 7ff86daec640 1 --2- 172.21.15.132:0/2032087853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681049a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff864009840 tx=0x7ff8640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37fa1f812c96e96 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86caea640 1 -- 172.21.15.132:0/2032087853 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff864032440 con 0x7ff868075b40 2024-06-13T01:18:49.942 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86caea640 1 -- 172.21.15.132:0/2032087853 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff864032a30 con 0x7ff868075b40 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86caea640 1 -- 172.21.15.132:0/2032087853 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff864031ad0 con 0x7ff868075b40 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 -- 172.21.15.132:0/2032087853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 msgr2=0x7ff8681049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 --2- 172.21.15.132:0/2032087853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681049a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff864009840 tx=0x7ff8640091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 -- 172.21.15.132:0/2032087853 shutdown_connections 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 --2- 172.21.15.132:0/2032087853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681049a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 -- 172.21.15.132:0/2032087853 >> 172.21.15.132:0/2032087853 conn(0x7ff8680fc430 msgr2=0x7ff8680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 -- 172.21.15.132:0/2032087853 shutdown_connections 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.197+0000 7ff86fd50640 1 -- 172.21.15.132:0/2032087853 wait complete. 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86fd50640 1 Processor -- start 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86fd50640 1 -- start start 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86fd50640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:49.943 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86fd50640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff868102810 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86daec640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86daec640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36142/0 (socket says 172.21.15.132:36142) 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86daec640 1 -- 172.21.15.132:0/3500672513 learned_addr learned my addr 172.21.15.132:0/3500672513 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.198+0000 7ff86daec640 1 -- 172.21.15.132:0/3500672513 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864009450 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff86daec640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681022d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff864006f00 tx=0x7ff864008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff86403d450 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff86403da40 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff86403bb10 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8681008a0 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff868100d10 con 0x7ff868075b40 2024-06-13T01:18:49.944 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.199+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff864039070 con 0x7ff868075b40 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.200+0000 7ff85e7fc640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.200+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff86403d5b0 con 0x7ff868075b40 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.200+0000 7ff86d2eb640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 msgr2=0x7ff854036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1802099589 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.200+0000 7ff86d2eb640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.200+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 --> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff86804b480 con 0x7ff854034810 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.401+0000 7ff86d2eb640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 msgr2=0x7ff854036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1802099589 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.401+0000 7ff86d2eb640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.802+0000 7ff86d2eb640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 msgr2=0x7ff854036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1802099589 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:46.802+0000 7ff86d2eb640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-13T01:18:49.945 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:47.603+0000 7ff86d2eb640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 msgr2=0x7ff854036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1802099589 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:47.603+0000 7ff86d2eb640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:48.375+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mgrmap(e 6) v1 ==== 41713+0+0 (secure 0 0 0) 0x7ff86403bd00 con 0x7ff868075b40 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:48.375+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 msgr2=0x7ff854036cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:48.375+0000 7ff85e7fc640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.375+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7ff86403aba0 con 0x7ff868075b40 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.376+0000 7ff85e7fc640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff854037fd0 0x7ff85403a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.376+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff86804b480 con 0x7ff854037fd0 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.376+0000 7ff86d2eb640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff854037fd0 0x7ff85403a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.377+0000 7ff86d2eb640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff854037fd0 0x7ff85403a3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff858003940 tx=0x7ff858009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.377+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7ff86804b480 con 0x7ff854037fd0 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.382+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff868105cc0 con 0x7ff854037fd0 2024-06-13T01:18:49.946 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff85e7fc640 1 -- 172.21.15.132:0/3500672513 <== mgr.14118 v2:172.21.15.132:6800/2668291289 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff868105cc0 con 0x7ff854037fd0 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff854037fd0 msgr2=0x7ff85403a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff854037fd0 0x7ff85403a3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff858003940 tx=0x7ff858009350 comp rx=0 tx=0).stop 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 msgr2=0x7ff8681022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681022d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff864006f00 tx=0x7ff864008040 comp rx=0 tx=0).stop 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 shutdown_connections 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff854037fd0 0x7ff85403a3b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:6800/1802099589,v1:172.21.15.132:6801/1802099589] conn(0x7ff854034810 0x7ff854036cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 --2- 172.21.15.132:0/3500672513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff868075b40 0x7ff8681022d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 >> 172.21.15.132:0/3500672513 conn(0x7ff8680fc430 msgr2=0x7ff8680fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 shutdown_connections 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:49.383+0000 7ff86fd50640 1 -- 172.21.15.132:0/3500672513 wait complete. 2024-06-13T01:18:49.947 INFO:teuthology.orchestra.run.smithi132.stdout:mgr epoch 5 is available 2024-06-13T01:18:49.948 INFO:teuthology.orchestra.run.smithi132.stdout:Setting orchestrator backend to cephadm... 2024-06-13T01:18:50.709 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:50 smithi132 ceph-mon[29330]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-13T01:18:50.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:50 smithi132 ceph-mon[29330]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-13T01:18:51.127 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.319+0000 7efd48457640 1 Processor -- start 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.319+0000 7efd48457640 1 -- start start 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.320+0000 7efd48457640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd401020e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.320+0000 7efd48457640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd40102620 con 0x7efd40101cd0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.320+0000 7efd461f3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd401020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.320+0000 7efd461f3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd401020e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36228/0 (socket says 172.21.15.132:36228) 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.320+0000 7efd461f3640 1 -- 172.21.15.132:0/3786044948 learned_addr learned my addr 172.21.15.132:0/3786044948 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.320+0000 7efd461f3640 1 -- 172.21.15.132:0/3786044948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40102760 con 0x7efd40101cd0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.321+0000 7efd461f3640 1 --2- 172.21.15.132:0/3786044948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd401020e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009840 tx=0x7efd3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e1a9e14b2af1dd server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.321+0000 7efd451f1640 1 -- 172.21.15.132:0/3786044948 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd3c032440 con 0x7efd40101cd0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.321+0000 7efd451f1640 1 -- 172.21.15.132:0/3786044948 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7efd3c032a30 con 0x7efd40101cd0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.321+0000 7efd451f1640 1 -- 172.21.15.132:0/3786044948 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd3c031ad0 con 0x7efd40101cd0 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- 172.21.15.132:0/3786044948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 msgr2=0x7efd401020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:51.128 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 --2- 172.21.15.132:0/3786044948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd401020e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009840 tx=0x7efd3c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- 172.21.15.132:0/3786044948 shutdown_connections 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 --2- 172.21.15.132:0/3786044948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd401020e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- 172.21.15.132:0/3786044948 >> 172.21.15.132:0/3786044948 conn(0x7efd400fd4b0 msgr2=0x7efd400ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- 172.21.15.132:0/3786044948 shutdown_connections 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- 172.21.15.132:0/3786044948 wait complete. 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 Processor -- start 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- start start 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd40110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd48457640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd40111010 con 0x7efd40101cd0 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd461f3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd40110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd461f3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd40110ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36236/0 (socket says 172.21.15.132:36236) 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.322+0000 7efd461f3640 1 -- 172.21.15.132:0/1479042145 learned_addr learned my addr 172.21.15.132:0/1479042145 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd461f3640 1 -- 172.21.15.132:0/1479042145 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c009450 con 0x7efd40101cd0 2024-06-13T01:18:51.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd461f3640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd40110ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009420 tx=0x7efd3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd3c03d5b0 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7efd3c03dba0 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd40111210 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd3c03bb40 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.323+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd40111680 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.324+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7efd3c039070 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.324+0000 7efd36ffd640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efd2c034730 0x7efd2c036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.324+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efd3c067480 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.324+0000 7efd459f2640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efd2c034730 0x7efd2c036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.324+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4005ead0 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.325+0000 7efd459f2640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efd2c034730 0x7efd2c036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efd30006fd0 tx=0x7efd30006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.329+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efd3c041020 con 0x7efd40101cd0 2024-06-13T01:18:51.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.427+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7efd3c038280 con 0x7efd40101cd0 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.538+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7efd400fe760 con 0x7efd2c034730 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.547+0000 7efd36ffd640 1 -- 172.21.15.132:0/1479042145 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efd400fe760 con 0x7efd2c034730 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efd2c034730 msgr2=0x7efd2c036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efd2c034730 0x7efd2c036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efd30006fd0 tx=0x7efd30006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 msgr2=0x7efd40110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd40110ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009420 tx=0x7efd3c008040 comp rx=0 tx=0).stop 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 shutdown_connections 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efd2c034730 0x7efd2c036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 --2- 172.21.15.132:0/1479042145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd40101cd0 0x7efd40110ad0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 >> 172.21.15.132:0/1479042145 conn(0x7efd400fd4b0 msgr2=0x7efd400fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 shutdown_connections 2024-06-13T01:18:51.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:50.549+0000 7efd48457640 1 -- 172.21.15.132:0/1479042145 wait complete. 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: mgrmap e8: smithi132.vjweoa(active, since 2s) 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: [13/Jun/2024:01:18:50] ENGINE Bus STARTING 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: [13/Jun/2024:01:18:50] ENGINE Serving on https://172.21.15.132:7150 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: [13/Jun/2024:01:18:50] ENGINE Bus STARTED 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: [13/Jun/2024:01:18:50] ENGINE Error in HTTPServer.serve 2024-06-13T01:18:51.679 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Traceback (most recent call last): 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._connections.run(self.expiration_interval) 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._run(expiration_interval) 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: s = self.context.wrap_socket( 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: return self.sslsocket_class._create( 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self.do_handshake() 2024-06-13T01:18:51.680 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T01:18:51.681 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._sslobj.do_handshake() 2024-06-13T01:18:51.681 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T01:18:51.681 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:51.681 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:51 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:18:52.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-13T01:18:52.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa133c6640 1 Processor -- start 2024-06-13T01:18:52.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa133c6640 1 -- start start 2024-06-13T01:18:52.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa133c6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c101670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:52.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa133c6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c101bb0 con 0x7ffa0c101260 2024-06-13T01:18:52.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa11162640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa11162640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c101670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36248/0 (socket says 172.21.15.132:36248) 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.507+0000 7ffa11162640 1 -- 172.21.15.132:0/679069460 learned_addr learned my addr 172.21.15.132:0/679069460 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.508+0000 7ffa11162640 1 -- 172.21.15.132:0/679069460 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c101cf0 con 0x7ffa0c101260 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.508+0000 7ffa11162640 1 --2- 172.21.15.132:0/679069460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c101670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800baf0 tx=0x7ffa0800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45731860f8bffce0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.508+0000 7ffa03fff640 1 -- 172.21.15.132:0/679069460 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa080327a0 con 0x7ffa0c101260 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa03fff640 1 -- 172.21.15.132:0/679069460 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffa08032d90 con 0x7ffa0c101260 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa03fff640 1 -- 172.21.15.132:0/679069460 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa08031e70 con 0x7ffa0c101260 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 -- 172.21.15.132:0/679069460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 msgr2=0x7ffa0c101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 --2- 172.21.15.132:0/679069460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c101670 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800baf0 tx=0x7ffa0800b460 comp rx=0 tx=0).stop 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 -- 172.21.15.132:0/679069460 shutdown_connections 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 --2- 172.21.15.132:0/679069460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c101670 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:52.202 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 -- 172.21.15.132:0/679069460 >> 172.21.15.132:0/679069460 conn(0x7ffa0c0fbbf0 msgr2=0x7ffa0c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 -- 172.21.15.132:0/679069460 shutdown_connections 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.509+0000 7ffa133c6640 1 -- 172.21.15.132:0/679069460 wait complete. 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa133c6640 1 Processor -- start 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa133c6640 1 -- start start 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa133c6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c10e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa133c6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c10eed0 con 0x7ffa0c101260 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa11162640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c10e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa11162640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c10e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:36252/0 (socket says 172.21.15.132:36252) 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa11162640 1 -- 172.21.15.132:0/1817653023 learned_addr learned my addr 172.21.15.132:0/1817653023 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa11162640 1 -- 172.21.15.132:0/1817653023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0800b700 con 0x7ffa0c101260 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.510+0000 7ffa11162640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c10e990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800b2c0 tx=0x7ffa0800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.511+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa08032a80 con 0x7ffa0c101260 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.511+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffa08032be0 con 0x7ffa0c101260 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.511+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0c10f0d0 con 0x7ffa0c101260 2024-06-13T01:18:52.203 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.511+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa08039750 con 0x7ffa0c101260 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.511+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0c10f4e0 con 0x7ffa0c101260 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.511+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ffa08039a10 con 0x7ffa0c101260 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.512+0000 7ffa01ffb640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff9f8034810 0x7ff9f8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.512+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ffa08069c00 con 0x7ffa0c101260 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.512+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0c04b480 con 0x7ffa0c101260 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.512+0000 7ffa10961640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff9f8034810 0x7ff9f8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.512+0000 7ffa10961640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff9f8034810 0x7ff9f8036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc006fd0 tx=0x7ff9fc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.516+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ffa08036030 con 0x7ffa0c101260 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.680+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa0c110090 con 0x7ff9f8034810 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.681+0000 7ffa01ffb640 1 -- 172.21.15.132:0/1817653023 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ffa0c110090 con 0x7ff9f8034810 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff9f8034810 msgr2=0x7ff9f8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff9f8034810 0x7ff9f8036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc006fd0 tx=0x7ff9fc006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:52.204 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 msgr2=0x7ffa0c10e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c10e990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800b2c0 tx=0x7ffa0800b460 comp rx=0 tx=0).stop 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 shutdown_connections 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff9f8034810 0x7ff9f8036cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 --2- 172.21.15.132:0/1817653023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa0c101260 0x7ffa0c10e990 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 >> 172.21.15.132:0/1817653023 conn(0x7ffa0c0fbbf0 msgr2=0x7ffa0c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 shutdown_connections 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:51.683+0000 7ffa133c6640 1 -- 172.21.15.132:0/1817653023 wait complete. 2024-06-13T01:18:52.205 INFO:teuthology.orchestra.run.smithi132.stdout:Generating ssh key... 2024-06-13T01:18:52.596 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:52 smithi132 ceph-mon[29330]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:18:53.457 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295b66e640 1 Processor -- start 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295b66e640 1 -- start start 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295b66e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f2954101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295b66e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29541015a0 con 0x7f2954100c50 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295940a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f2954101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295940a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f2954101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:42968/0 (socket says 172.21.15.132:42968) 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295940a640 1 -- 172.21.15.132:0/4037554164 learned_addr learned my addr 172.21.15.132:0/4037554164 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.598+0000 7f295940a640 1 -- 172.21.15.132:0/4037554164 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29541016e0 con 0x7f2954100c50 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.599+0000 7f295940a640 1 --2- 172.21.15.132:0/4037554164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f2954101060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d0d0 tx=0x7f294c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9efbd74ad56903 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:53.458 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.599+0000 7f2953fff640 1 -- 172.21.15.132:0/4037554164 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c00b440 con 0x7f2954100c50 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.599+0000 7f2953fff640 1 -- 172.21.15.132:0/4037554164 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f294c009aa0 con 0x7f2954100c50 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.599+0000 7f2953fff640 1 -- 172.21.15.132:0/4037554164 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c032bc0 con 0x7f2954100c50 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 -- 172.21.15.132:0/4037554164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 msgr2=0x7f2954101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 --2- 172.21.15.132:0/4037554164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f2954101060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d0d0 tx=0x7f294c00ba50 comp rx=0 tx=0).stop 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 -- 172.21.15.132:0/4037554164 shutdown_connections 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 --2- 172.21.15.132:0/4037554164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f2954101060 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 -- 172.21.15.132:0/4037554164 >> 172.21.15.132:0/4037554164 conn(0x7f29540fc430 msgr2=0x7f29540fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 -- 172.21.15.132:0/4037554164 shutdown_connections 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 -- 172.21.15.132:0/4037554164 wait complete. 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 Processor -- start 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.600+0000 7f295b66e640 1 -- start start 2024-06-13T01:18:53.459 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295b66e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f295410e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295b66e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295410ee40 con 0x7f2954100c50 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295940a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f295410e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295940a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f295410e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:42972/0 (socket says 172.21.15.132:42972) 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295940a640 1 -- 172.21.15.132:0/3559367146 learned_addr learned my addr 172.21.15.132:0/3559367146 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295940a640 1 -- 172.21.15.132:0/3559367146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c00bcf0 con 0x7f2954100c50 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295940a640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f295410e900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d130 tx=0x7f294c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c0453f0 con 0x7f2954100c50 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f294c0459e0 con 0x7f2954100c50 2024-06-13T01:18:53.460 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295410f040 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c03ba30 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.601+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295410fb80 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.602+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f294c03bcf0 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.602+0000 7f2951ffb640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f293c034810 0x7f293c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.602+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f294c06a160 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.602+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295405ead0 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.603+0000 7f2958c09640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f293c034810 0x7f293c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.603+0000 7f2958c09640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f293c034810 0x7f293c036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2948006fd0 tx=0x7f2948006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.607+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f294c04d390 con 0x7f2954100c50 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.773+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f295410fed0 con 0x7f293c034810 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.853+0000 7f2951ffb640 1 -- 172.21.15.132:0/3559367146 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f295410fed0 con 0x7f293c034810 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.855+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f293c034810 msgr2=0x7f293c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:53.461 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f293c034810 0x7f293c036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2948006fd0 tx=0x7f2948006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 msgr2=0x7f295410e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f295410e900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d130 tx=0x7f294c00ba50 comp rx=0 tx=0).stop 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 shutdown_connections 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f293c034810 0x7f293c036cc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 --2- 172.21.15.132:0/3559367146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2954100c50 0x7f295410e900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 >> 172.21.15.132:0/3559367146 conn(0x7f29540fc430 msgr2=0x7f2954105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 shutdown_connections 2024-06-13T01:18:53.462 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:52.856+0000 7f295b66e640 1 -- 172.21.15.132:0/3559367146 wait complete. 2024-06-13T01:18:54.013 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:53 smithi132 ceph-mon[29330]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:18:54.014 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:53 smithi132 ceph-mon[29330]: Generating ssh key... 2024-06-13T01:18:54.014 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:53 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:54.014 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:53 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7143o7mu/808ba+eIYaYptqQZdlOukJNvzUPJt4nxXnvOX8IBXhy18cG1mWW7DW4kzdjW/8/g5MDVzm3S+JdDgv3a4ODksH7hO92qTdKQww+VMaI1BK2TSatOOpoVz2aaoiuOEf5AQ+lDAHnPAWzPLkAy2/PSmf9x+8X3nRFjDOe9R524T7KPqufLhcbzehHeGnyhT01ee1TKQJ/ANkJGtOszyu7p45JpxUETFCYeNdgwMoOjsyMebKiY3g2Nubt5SEAo77AAUT6nWlEr2E+Gul0uM6QALzoit8fpcmlm0GdAFfGn3UP/p/I6ZRAJWUsQH01t/zoPITl0moyOjXTWa9ILhi0NrKxp9UhBey/QLEerf7UL2CBKjIg5QIif4kSB3L2NWWpvXmJl93ZswXOnEB/7UrErptd1HiIpRpozLjYWEX7qe/0Y0bYgSgVUuAPs+x7xQQ+rLygnrfNCDEVyP0x4ZT6X5b0i/K4JOQHgf5LfuqWavrLYE4xshXLcCEE= ceph-ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72c2bf640 1 Processor -- start 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72c2bf640 1 -- start start 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72c2bf640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff7241049c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72c2bf640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff724075f60 con 0x7ff724075b50 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72a05b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff7241049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72a05b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff7241049c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:42986/0 (socket says 172.21.15.132:42986) 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72a05b640 1 -- 172.21.15.132:0/1405957861 learned_addr learned my addr 172.21.15.132:0/1405957861 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.838+0000 7ff72a05b640 1 -- 172.21.15.132:0/1405957861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff724104f00 con 0x7ff724075b50 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.839+0000 7ff72a05b640 1 --2- 172.21.15.132:0/1405957861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff7241049c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff720009840 tx=0x7ff7200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648fa17e912a1976 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:54.640 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.839+0000 7ff729059640 1 -- 172.21.15.132:0/1405957861 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff720032440 con 0x7ff724075b50 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.839+0000 7ff729059640 1 -- 172.21.15.132:0/1405957861 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff720032a30 con 0x7ff724075b50 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.839+0000 7ff729059640 1 -- 172.21.15.132:0/1405957861 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff720031ad0 con 0x7ff724075b50 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 -- 172.21.15.132:0/1405957861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 msgr2=0x7ff7241049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 --2- 172.21.15.132:0/1405957861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff7241049c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff720009840 tx=0x7ff7200091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 -- 172.21.15.132:0/1405957861 shutdown_connections 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 --2- 172.21.15.132:0/1405957861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff7241049c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 -- 172.21.15.132:0/1405957861 >> 172.21.15.132:0/1405957861 conn(0x7ff7240fc4b0 msgr2=0x7ff7240fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 -- 172.21.15.132:0/1405957861 shutdown_connections 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 -- 172.21.15.132:0/1405957861 wait complete. 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.840+0000 7ff72c2bf640 1 Processor -- start 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72c2bf640 1 -- start start 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72c2bf640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff724110ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:54.641 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72c2bf640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff724111000 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72a05b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff724110ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72a05b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff724110ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:42996/0 (socket says 172.21.15.132:42996) 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72a05b640 1 -- 172.21.15.132:0/519446665 learned_addr learned my addr 172.21.15.132:0/519446665 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72a05b640 1 -- 172.21.15.132:0/519446665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720009450 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72a05b640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff724110ac0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff720031b90 tx=0x7ff720008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7200328b0 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff724111200 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff72003d8e0 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff72003ba20 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.841+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff724111670 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.842+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff720039070 con 0x7ff724075b50 2024-06-13T01:18:54.642 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.842+0000 7ff71affd640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff710034810 0x7ff710036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.842+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff7200676a0 con 0x7ff724075b50 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.842+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72405ead0 con 0x7ff724075b50 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.842+0000 7ff72985a640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff710034810 0x7ff710036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.843+0000 7ff72985a640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff710034810 0x7ff710036cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff714006fd0 tx=0x7ff714006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:53.847+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff720041020 con 0x7ff724075b50 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.015+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7240fd700 con 0x7ff710034810 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.016+0000 7ff71affd640 1 -- 172.21.15.132:0/519446665 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff7240fd700 con 0x7ff710034810 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff710034810 msgr2=0x7ff710036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff710034810 0x7ff710036cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff714006fd0 tx=0x7ff714006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 msgr2=0x7ff724110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff724110ac0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff720031b90 tx=0x7ff720008040 comp rx=0 tx=0).stop 2024-06-13T01:18:54.643 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 shutdown_connections 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7ff710034810 0x7ff710036cc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 --2- 172.21.15.132:0/519446665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724075b50 0x7ff724110ac0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 >> 172.21.15.132:0/519446665 conn(0x7ff7240fc4b0 msgr2=0x7ff7240fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.018+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 shutdown_connections 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:54.019+0000 7ff72c2bf640 1 -- 172.21.15.132:0/519446665 wait complete. 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:Adding key to root@localhost authorized_keys... 2024-06-13T01:18:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:Adding host smithi132... 2024-06-13T01:18:55.189 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:55 smithi132 ceph-mon[29330]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:18:56.413 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:56 smithi132 ceph-mon[29330]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "addr": "172.21.15.132", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:18:57.311 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:57 smithi132 ceph-mon[29330]: Deploying cephadm binary to smithi132 2024-06-13T01:18:58.465 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Added host 'smithi132' with addr '172.21.15.132' 2024-06-13T01:18:58.465 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb404e96640 1 Processor -- start 2024-06-13T01:18:58.465 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb404e96640 1 -- start start 2024-06-13T01:18:58.465 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb404e96640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb404e96640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb400075f50 con 0x7fb400075b40 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb3fe59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb3fe59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43006/0 (socket says 172.21.15.132:43006) 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.014+0000 7fb3fe59c640 1 -- 172.21.15.132:0/3463432686 learned_addr learned my addr 172.21.15.132:0/3463432686 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.015+0000 7fb3fe59c640 1 -- 172.21.15.132:0/3463432686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb400102790 con 0x7fb400075b40 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.015+0000 7fb3fe59c640 1 --2- 172.21.15.132:0/3463432686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400106da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009840 tx=0x7fb3f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374e3619024b2a41 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.015+0000 7fb3fdd9b640 1 -- 172.21.15.132:0/3463432686 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f0032440 con 0x7fb400075b40 2024-06-13T01:18:58.466 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.015+0000 7fb3fdd9b640 1 -- 172.21.15.132:0/3463432686 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb3f0032a30 con 0x7fb400075b40 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.015+0000 7fb3fdd9b640 1 -- 172.21.15.132:0/3463432686 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f0031ad0 con 0x7fb400075b40 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 -- 172.21.15.132:0/3463432686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 msgr2=0x7fb400106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 --2- 172.21.15.132:0/3463432686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400106da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009840 tx=0x7fb3f00091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 -- 172.21.15.132:0/3463432686 shutdown_connections 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 --2- 172.21.15.132:0/3463432686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400106da0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 -- 172.21.15.132:0/3463432686 >> 172.21.15.132:0/3463432686 conn(0x7fb4000fc430 msgr2=0x7fb4000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:58.469 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 -- 172.21.15.132:0/3463432686 shutdown_connections 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 -- 172.21.15.132:0/3463432686 wait complete. 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 Processor -- start 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.016+0000 7fb404e96640 1 -- start start 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb404e96640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400112c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb404e96640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4001131d0 con 0x7fb400075b40 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3fe59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400112c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3fe59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400112c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43014/0 (socket says 172.21.15.132:43014) 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3fe59c640 1 -- 172.21.15.132:0/4192231644 learned_addr learned my addr 172.21.15.132:0/4192231644 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3fe59c640 1 -- 172.21.15.132:0/4192231644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0009450 con 0x7fb400075b40 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3fe59c640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400112c90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0006f00 tx=0x7fb3f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:58.470 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f003d450 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb3f003da40 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb400113430 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f003bb10 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.017+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb400113f60 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.018+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb3f0039070 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.018+0000 7fb3f6ffd640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fb3ec034810 0x7fb3ec036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.018+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40005ead0 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.018+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb3f0067860 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.018+0000 7fb3f7fff640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fb3ec034810 0x7fb3ec036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.019+0000 7fb3f7fff640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fb3ec034810 0x7fb3ec036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8006fd0 tx=0x7fb3e8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.023+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb3f0041080 con 0x7fb400075b40 2024-06-13T01:18:58.471 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:55.191+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi132", "addr": "172.21.15.132", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4000fd710 con 0x7fb3ec034810 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.878+0000 7fb3f6ffd640 1 -- 172.21.15.132:0/4192231644 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb4000fd710 con 0x7fb3ec034810 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fb3ec034810 msgr2=0x7fb3ec036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fb3ec034810 0x7fb3ec036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8006fd0 tx=0x7fb3e8006db0 comp rx=0 tx=0).stop 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 msgr2=0x7fb400112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400112c90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0006f00 tx=0x7fb3f0008040 comp rx=0 tx=0).stop 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 shutdown_connections 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fb3ec034810 0x7fb3ec036cc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 --2- 172.21.15.132:0/4192231644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb400075b40 0x7fb400112c90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 >> 172.21.15.132:0/4192231644 conn(0x7fb4000fc430 msgr2=0x7fb4000fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 shutdown_connections 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:57.881+0000 7fb404e96640 1 -- 172.21.15.132:0/4192231644 wait complete. 2024-06-13T01:18:58.472 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying mon service with default placement... 2024-06-13T01:18:59.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:58 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:18:59.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:58 smithi132 ceph-mon[29330]: Added host smithi132 2024-06-13T01:18:59.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:18:58 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:18:59.890 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57edc0640 1 Processor -- start 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57edc0640 1 -- start start 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57edc0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57edc0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa578101b20 con 0x7fa5781011d0 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57cb5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57cb5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43026/0 (socket says 172.21.15.132:43026) 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57cb5c640 1 -- 172.21.15.132:0/182033585 learned_addr learned my addr 172.21.15.132:0/182033585 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.891+0000 7fa57cb5c640 1 -- 172.21.15.132:0/182033585 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578101c60 con 0x7fa5781011d0 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.892+0000 7fa57cb5c640 1 --2- 172.21.15.132:0/182033585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781015e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa570009840 tx=0x7fa5700091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e9885016451ee9 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.892+0000 7fa5777fe640 1 -- 172.21.15.132:0/182033585 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa570032440 con 0x7fa5781011d0 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.892+0000 7fa5777fe640 1 -- 172.21.15.132:0/182033585 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa570032a30 con 0x7fa5781011d0 2024-06-13T01:18:59.891 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.892+0000 7fa5777fe640 1 -- 172.21.15.132:0/182033585 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa570031ad0 con 0x7fa5781011d0 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 -- 172.21.15.132:0/182033585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 msgr2=0x7fa5781015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 --2- 172.21.15.132:0/182033585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781015e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa570009840 tx=0x7fa5700091d0 comp rx=0 tx=0).stop 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 -- 172.21.15.132:0/182033585 shutdown_connections 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 --2- 172.21.15.132:0/182033585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781015e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 -- 172.21.15.132:0/182033585 >> 172.21.15.132:0/182033585 conn(0x7fa5780fbbc0 msgr2=0x7fa5780fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 -- 172.21.15.132:0/182033585 shutdown_connections 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 -- 172.21.15.132:0/182033585 wait complete. 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 Processor -- start 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.893+0000 7fa57edc0640 1 -- start start 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57edc0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57edc0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781115e0 con 0x7fa5781011d0 2024-06-13T01:18:59.892 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57cb5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57cb5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43030/0 (socket says 172.21.15.132:43030) 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57cb5c640 1 -- 172.21.15.132:0/1908332678 learned_addr learned my addr 172.21.15.132:0/1908332678 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57cb5c640 1 -- 172.21.15.132:0/1908332678 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570009450 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57cb5c640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781009d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa5700317b0 tx=0x7fa570008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa570031e90 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa57003daf0 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa578115bf0 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa57003baa0 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.894+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa578115f20 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.895+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fa570039070 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.895+0000 7fa5757fa640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa564034810 0x7fa564036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.895+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa5700678b0 con 0x7fa5781011d0 2024-06-13T01:18:59.893 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.895+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57804b480 con 0x7fa5781011d0 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.895+0000 7fa577fff640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa564034810 0x7fa564036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.896+0000 7fa577fff640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa564034810 0x7fa564036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa56800ad90 tx=0x7fa568009410 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:58.900+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa570041080 con 0x7fa5781011d0 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.112+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa578116160 con 0x7fa564034810 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.119+0000 7fa5757fa640 1 -- 172.21.15.132:0/1908332678 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa578116160 con 0x7fa564034810 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa564034810 msgr2=0x7fa564036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa564034810 0x7fa564036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa56800ad90 tx=0x7fa568009410 comp rx=0 tx=0).stop 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 msgr2=0x7fa5781009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781009d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa5700317b0 tx=0x7fa570008040 comp rx=0 tx=0).stop 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 shutdown_connections 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa564034810 0x7fa564036cc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 --2- 172.21.15.132:0/1908332678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5781011d0 0x7fa5781009d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 >> 172.21.15.132:0/1908332678 conn(0x7fa5780fbbc0 msgr2=0x7fa5780fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:18:59.894 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 shutdown_connections 2024-06-13T01:18:59.895 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:18:59.121+0000 7fa57edc0640 1 -- 172.21.15.132:0/1908332678 wait complete. 2024-06-13T01:18:59.895 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying mgr service with default placement... 2024-06-13T01:19:00.209 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:00 smithi132 ceph-mon[29330]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:00.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:00 smithi132 ceph-mon[29330]: Saving service mon spec with placement count:5 2024-06-13T01:19:00.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:00 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:02.087 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-13T01:19:02.087 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.320+0000 7fc175a8c640 1 Processor -- start 2024-06-13T01:19:02.087 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc175a8c640 1 -- start start 2024-06-13T01:19:02.087 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc175a8c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170103770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:02.087 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc175a8c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700708a0 con 0x7fc170103360 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43040/0 (socket says 172.21.15.132:43040) 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc16f7fe640 1 -- 172.21.15.132:0/1285013090 learned_addr learned my addr 172.21.15.132:0/1285013090 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.321+0000 7fc16f7fe640 1 -- 172.21.15.132:0/1285013090 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170103cb0 con 0x7fc170103360 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.322+0000 7fc16f7fe640 1 --2- 172.21.15.132:0/1285013090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170103770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc160009840 tx=0x7fc1600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac102d0c03c82fb server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc16e7fc640 1 -- 172.21.15.132:0/1285013090 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc160032440 con 0x7fc170103360 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc16e7fc640 1 -- 172.21.15.132:0/1285013090 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc160032a70 con 0x7fc170103360 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc16e7fc640 1 -- 172.21.15.132:0/1285013090 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc160031ad0 con 0x7fc170103360 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 -- 172.21.15.132:0/1285013090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 msgr2=0x7fc170103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 --2- 172.21.15.132:0/1285013090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170103770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc160009840 tx=0x7fc1600091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 -- 172.21.15.132:0/1285013090 shutdown_connections 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 --2- 172.21.15.132:0/1285013090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170103770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:02.088 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 -- 172.21.15.132:0/1285013090 >> 172.21.15.132:0/1285013090 conn(0x7fc1700fbbc0 msgr2=0x7fc1700fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 -- 172.21.15.132:0/1285013090 shutdown_connections 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.323+0000 7fc175a8c640 1 -- 172.21.15.132:0/1285013090 wait complete. 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc175a8c640 1 Processor -- start 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc175a8c640 1 -- start start 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc175a8c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc175a8c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170111110 con 0x7fc170103360 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43048/0 (socket says 172.21.15.132:43048) 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.324+0000 7fc16f7fe640 1 -- 172.21.15.132:0/1347479991 learned_addr learned my addr 172.21.15.132:0/1347479991 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc16f7fe640 1 -- 172.21.15.132:0/1347479991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160009450 con 0x7fc170103360 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc16f7fe640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170110bd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc160006a70 tx=0x7fc16002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc160031e90 con 0x7fc170103360 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc170111310 con 0x7fc170103360 2024-06-13T01:19:02.089 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc16003b910 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc170111780 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.325+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc16003a940 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.326+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fc160039070 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.326+0000 7fc157fff640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fc1580348b0 0x7fc158036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.326+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc160067150 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.326+0000 7fc16effd640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fc1580348b0 0x7fc158036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.327+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17005ead0 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.327+0000 7fc16effd640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fc1580348b0 0x7fc158036d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc164006fd0 tx=0x7fc164006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.332+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc160037030 con 0x7fc170103360 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.515+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1700fce90 con 0x7fc1580348b0 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.523+0000 7fc157fff640 1 -- 172.21.15.132:0/1347479991 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc1700fce90 con 0x7fc1580348b0 2024-06-13T01:19:02.090 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fc1580348b0 msgr2=0x7fc158036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fc1580348b0 0x7fc158036d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc164006fd0 tx=0x7fc164006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 msgr2=0x7fc170110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170110bd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc160006a70 tx=0x7fc16002c040 comp rx=0 tx=0).stop 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 shutdown_connections 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fc1580348b0 0x7fc158036d60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 --2- 172.21.15.132:0/1347479991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc170103360 0x7fc170110bd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 >> 172.21.15.132:0/1347479991 conn(0x7fc1700fbbc0 msgr2=0x7fc1700fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 shutdown_connections 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:01.525+0000 7fc175a8c640 1 -- 172.21.15.132:0/1347479991 wait complete. 2024-06-13T01:19:02.091 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying crash service with default placement... 2024-06-13T01:19:02.374 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:02 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:02.375 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:02 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:02.375 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:02 smithi132 ceph-mon[29330]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:02.375 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:02 smithi132 ceph-mon[29330]: Saving service mgr spec with placement count:2 2024-06-13T01:19:02.375 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:02 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:03.244 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-13T01:19:03.244 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f17219dc640 1 Processor -- start 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f17219dc640 1 -- start start 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f17219dc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c100590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f17219dc640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c100ad0 con 0x7f171c100180 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f171affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f171affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c100590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40714/0 (socket says 172.21.15.132:40714) 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f171affd640 1 -- 172.21.15.132:0/697515040 learned_addr learned my addr 172.21.15.132:0/697515040 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.450+0000 7f171affd640 1 -- 172.21.15.132:0/697515040 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c100c10 con 0x7f171c100180 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.451+0000 7f171affd640 1 --2- 172.21.15.132:0/697515040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c100590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f170c009840 tx=0x7f170c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790aeb5a1398da6b server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.451+0000 7f1719ffb640 1 -- 172.21.15.132:0/697515040 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170c032440 con 0x7f171c100180 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.451+0000 7f1719ffb640 1 -- 172.21.15.132:0/697515040 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f170c032a70 con 0x7f171c100180 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.451+0000 7f1719ffb640 1 -- 172.21.15.132:0/697515040 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170c031ad0 con 0x7f171c100180 2024-06-13T01:19:03.245 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 -- 172.21.15.132:0/697515040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 msgr2=0x7f171c100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 --2- 172.21.15.132:0/697515040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c100590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f170c009840 tx=0x7f170c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 -- 172.21.15.132:0/697515040 shutdown_connections 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 --2- 172.21.15.132:0/697515040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c100590 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 -- 172.21.15.132:0/697515040 >> 172.21.15.132:0/697515040 conn(0x7f171c0fbbc0 msgr2=0x7f171c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 -- 172.21.15.132:0/697515040 shutdown_connections 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 -- 172.21.15.132:0/697515040 wait complete. 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 Processor -- start 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.452+0000 7f17219dc640 1 -- start start 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17219dc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c110ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17219dc640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c1110e0 con 0x7f171c100180 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f171affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c110ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:03.246 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f171affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c110ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40728/0 (socket says 172.21.15.132:40728) 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f171affd640 1 -- 172.21.15.132:0/2150596556 learned_addr learned my addr 172.21.15.132:0/2150596556 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f171affd640 1 -- 172.21.15.132:0/2150596556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c009450 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f171affd640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c110ba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f170c009810 tx=0x7f170c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170c0328b0 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f170c03b910 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c1112e0 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170c03a980 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.453+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c111750 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.454+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f170c039070 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.454+0000 7f17037fe640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1704034860 0x7f1704036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.454+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f170c066df0 con 0x7f171c100180 2024-06-13T01:19:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.454+0000 7f171a7fc640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1704034860 0x7f1704036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.454+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c05ead0 con 0x7f171c100180 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.455+0000 7f171a7fc640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1704034860 0x7f1704036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1710006fd0 tx=0x7f1710006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.459+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f170c037070 con 0x7f171c100180 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.656+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f171c0fcf70 con 0x7f1704034860 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.660+0000 7f17037fe640 1 -- 172.21.15.132:0/2150596556 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f171c0fcf70 con 0x7f1704034860 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1704034860 msgr2=0x7f1704036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1704034860 0x7f1704036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1710006fd0 tx=0x7f1710006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 msgr2=0x7f171c110ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c110ba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f170c009810 tx=0x7f170c02c040 comp rx=0 tx=0).stop 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 shutdown_connections 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1704034860 0x7f1704036d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 --2- 172.21.15.132:0/2150596556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f171c100180 0x7f171c110ba0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 >> 172.21.15.132:0/2150596556 conn(0x7f171c0fbbc0 msgr2=0x7f171c0fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:03.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 shutdown_connections 2024-06-13T01:19:03.249 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:02.663+0000 7f17219dc640 1 -- 172.21.15.132:0/2150596556 wait complete. 2024-06-13T01:19:03.249 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying prometheus service with default placement... 2024-06-13T01:19:03.751 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:03 smithi132 ceph-mon[29330]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:03.751 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:03 smithi132 ceph-mon[29330]: Saving service crash spec with placement * 2024-06-13T01:19:03.752 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:03 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:04.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a56b5c640 1 Processor -- start 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a56b5c640 1 -- start start 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a56b5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a501049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a56b5c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a50075f50 con 0x7f3a50075b40 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a548f8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a501049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a548f8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a501049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40740/0 (socket says 172.21.15.132:40740) 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a548f8640 1 -- 172.21.15.132:0/4256282672 learned_addr learned my addr 172.21.15.132:0/4256282672 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.676+0000 7f3a548f8640 1 -- 172.21.15.132:0/4256282672 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a50104ee0 con 0x7f3a50075b40 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.677+0000 7f3a548f8640 1 --2- 172.21.15.132:0/4256282672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a501049a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009840 tx=0x7f3a440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f5e23089d49bf6 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:04.611 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.677+0000 7f3a4f7fe640 1 -- 172.21.15.132:0/4256282672 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a44032440 con 0x7f3a50075b40 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.677+0000 7f3a4f7fe640 1 -- 172.21.15.132:0/4256282672 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a44032a70 con 0x7f3a50075b40 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.677+0000 7f3a4f7fe640 1 -- 172.21.15.132:0/4256282672 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a44031ad0 con 0x7f3a50075b40 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 -- 172.21.15.132:0/4256282672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 msgr2=0x7f3a501049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 --2- 172.21.15.132:0/4256282672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a501049a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009840 tx=0x7f3a440091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 -- 172.21.15.132:0/4256282672 shutdown_connections 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 --2- 172.21.15.132:0/4256282672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a501049a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 -- 172.21.15.132:0/4256282672 >> 172.21.15.132:0/4256282672 conn(0x7f3a500fc430 msgr2=0x7f3a500fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 -- 172.21.15.132:0/4256282672 shutdown_connections 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 -- 172.21.15.132:0/4256282672 wait complete. 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 Processor -- start 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.678+0000 7f3a56b5c640 1 -- start start 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a56b5c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a50110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a56b5c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a50111110 con 0x7f3a50075b40 2024-06-13T01:19:04.612 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a548f8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a50110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a548f8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a50110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40756/0 (socket says 172.21.15.132:40756) 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a548f8640 1 -- 172.21.15.132:0/849581255 learned_addr learned my addr 172.21.15.132:0/849581255 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a548f8640 1 -- 172.21.15.132:0/849581255 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a44009450 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a548f8640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a50110bd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3a440317f0 tx=0x7f3a4402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.679+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a44032900 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.680+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a4403b910 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.680+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a50111310 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.680+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a4403a960 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.680+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a50111780 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.681+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3a44039070 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.681+0000 7f3a4d7fa640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f3a40034810 0x7f3a40036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.681+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5004b480 con 0x7f3a50075b40 2024-06-13T01:19:04.613 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.681+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3a44066cd0 con 0x7f3a50075b40 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.681+0000 7f3a4ffff640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f3a40034810 0x7f3a40036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.681+0000 7f3a4ffff640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f3a40034810 0x7f3a40036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c006fd0 tx=0x7f3a3c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.685+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3a44030750 con 0x7f3a50075b40 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.871+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a50111780 con 0x7f3a40034810 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.875+0000 7f3a4d7fa640 1 -- 172.21.15.132:0/849581255 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f3a50111780 con 0x7f3a40034810 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f3a40034810 msgr2=0x7f3a40036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f3a40034810 0x7f3a40036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c006fd0 tx=0x7f3a3c006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 msgr2=0x7f3a50110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a50110bd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3a440317f0 tx=0x7f3a4402c040 comp rx=0 tx=0).stop 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 shutdown_connections 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f3a40034810 0x7f3a40036cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 --2- 172.21.15.132:0/849581255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a50075b40 0x7f3a50110bd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 >> 172.21.15.132:0/849581255 conn(0x7f3a500fc430 msgr2=0x7f3a500fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:04.614 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 shutdown_connections 2024-06-13T01:19:04.615 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:03.878+0000 7f3a56b5c640 1 -- 172.21.15.132:0/849581255 wait complete. 2024-06-13T01:19:04.615 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying grafana service with default placement... 2024-06-13T01:19:04.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:04 smithi132 ceph-mon[29330]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:04.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:04 smithi132 ceph-mon[29330]: Saving service prometheus spec with placement count:1 2024-06-13T01:19:04.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:04 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:05.971 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-13T01:19:05.971 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.986+0000 7fa8773d5640 1 Processor -- start 2024-06-13T01:19:05.971 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.986+0000 7fa8773d5640 1 -- start start 2024-06-13T01:19:05.971 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.987+0000 7fa8773d5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.987+0000 7fa8773d5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700780d0 con 0x7fa870077cc0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.987+0000 7fa875171640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.987+0000 7fa875171640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40772/0 (socket says 172.21.15.132:40772) 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.987+0000 7fa875171640 1 -- 172.21.15.132:0/2239192067 learned_addr learned my addr 172.21.15.132:0/2239192067 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.987+0000 7fa875171640 1 -- 172.21.15.132:0/2239192067 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870076780 con 0x7fa870077cc0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.988+0000 7fa875171640 1 --2- 172.21.15.132:0/2239192067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870076240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009840 tx=0x7fa86c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c32e24b30a5ce7 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.988+0000 7fa863fff640 1 -- 172.21.15.132:0/2239192067 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c032440 con 0x7fa870077cc0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.988+0000 7fa863fff640 1 -- 172.21.15.132:0/2239192067 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa86c032a70 con 0x7fa870077cc0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.988+0000 7fa863fff640 1 -- 172.21.15.132:0/2239192067 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c031ad0 con 0x7fa870077cc0 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- 172.21.15.132:0/2239192067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 msgr2=0x7fa870076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:05.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 --2- 172.21.15.132:0/2239192067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870076240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009840 tx=0x7fa86c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- 172.21.15.132:0/2239192067 shutdown_connections 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 --2- 172.21.15.132:0/2239192067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870076240 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- 172.21.15.132:0/2239192067 >> 172.21.15.132:0/2239192067 conn(0x7fa8700fd490 msgr2=0x7fa8700ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- 172.21.15.132:0/2239192067 shutdown_connections 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- 172.21.15.132:0/2239192067 wait complete. 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 Processor -- start 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- start start 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.989+0000 7fa8773d5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa870113390 con 0x7fa870077cc0 2024-06-13T01:19:05.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa875171640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa875171640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870112e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40778/0 (socket says 172.21.15.132:40778) 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa875171640 1 -- 172.21.15.132:0/4145585941 learned_addr learned my addr 172.21.15.132:0/4145585941 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa875171640 1 -- 172.21.15.132:0/4145585941 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c009450 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa875171640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870112e50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0317f0 tx=0x7fa86c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c032900 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa86c03b910 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa870113590 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c03a960 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.990+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa870113fe0 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.991+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fa86c039070 con 0x7fa870077cc0 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.991+0000 7fa861ffb640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa85c034860 0x7fa85c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:05.974 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.991+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa86c0671c0 con 0x7fa870077cc0 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.991+0000 7fa874970640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa85c034860 0x7fa85c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.991+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87005ead0 con 0x7fa870077cc0 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.992+0000 7fa874970640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa85c034860 0x7fa85c036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa864006fd0 tx=0x7fa864006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:04.996+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa86c037030 con 0x7fa870077cc0 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.171+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8700fe850 con 0x7fa85c034860 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.175+0000 7fa861ffb640 1 -- 172.21.15.132:0/4145585941 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa8700fe850 con 0x7fa85c034860 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa85c034860 msgr2=0x7fa85c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa85c034860 0x7fa85c036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa864006fd0 tx=0x7fa864006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 msgr2=0x7fa870112e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870112e50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0317f0 tx=0x7fa86c02c040 comp rx=0 tx=0).stop 2024-06-13T01:19:05.975 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 shutdown_connections 2024-06-13T01:19:05.976 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fa85c034860 0x7fa85c036d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:05.976 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 --2- 172.21.15.132:0/4145585941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa870077cc0 0x7fa870112e50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:05.976 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 >> 172.21.15.132:0/4145585941 conn(0x7fa8700fd490 msgr2=0x7fa8700fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:05.976 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 shutdown_connections 2024-06-13T01:19:05.976 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:05.177+0000 7fa8773d5640 1 -- 172.21.15.132:0/4145585941 wait complete. 2024-06-13T01:19:05.976 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying node-exporter service with default placement... 2024-06-13T01:19:06.215 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:06 smithi132 ceph-mon[29330]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:06.215 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:06 smithi132 ceph-mon[29330]: Saving service grafana spec with placement count:1 2024-06-13T01:19:06.215 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:06 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:06.215 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:06 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.378+0000 7f1acba59640 1 Processor -- start 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.378+0000 7f1acba59640 1 -- start start 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.378+0000 7f1acba59640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac41010c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.378+0000 7f1acba59640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4101600 con 0x7f1ac4100cb0 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.379+0000 7f1ac97f5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac41010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.379+0000 7f1ac97f5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac41010c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40790/0 (socket says 172.21.15.132:40790) 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.379+0000 7f1ac97f5640 1 -- 172.21.15.132:0/2679905531 learned_addr learned my addr 172.21.15.132:0/2679905531 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.379+0000 7f1ac97f5640 1 -- 172.21.15.132:0/2679905531 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4101740 con 0x7f1ac4100cb0 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.379+0000 7f1ac97f5640 1 --2- 172.21.15.132:0/2679905531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac41010c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009840 tx=0x7f1ac00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c8f83d7920a85 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:07.194 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1abbfff640 1 -- 172.21.15.132:0/2679905531 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac0032440 con 0x7f1ac4100cb0 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1abbfff640 1 -- 172.21.15.132:0/2679905531 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac0032a70 con 0x7f1ac4100cb0 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1abbfff640 1 -- 172.21.15.132:0/2679905531 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac0031ad0 con 0x7f1ac4100cb0 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 -- 172.21.15.132:0/2679905531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 msgr2=0x7f1ac41010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 --2- 172.21.15.132:0/2679905531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac41010c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009840 tx=0x7f1ac00091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 -- 172.21.15.132:0/2679905531 shutdown_connections 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 --2- 172.21.15.132:0/2679905531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac41010c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 -- 172.21.15.132:0/2679905531 >> 172.21.15.132:0/2679905531 conn(0x7f1ac40fc490 msgr2=0x7f1ac40fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 -- 172.21.15.132:0/2679905531 shutdown_connections 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.380+0000 7f1acba59640 1 -- 172.21.15.132:0/2679905531 wait complete. 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1acba59640 1 Processor -- start 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1acba59640 1 -- start start 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1acba59640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac4110b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:07.195 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1acba59640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac41110a0 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1ac97f5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac4110b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1ac97f5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac4110b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40806/0 (socket says 172.21.15.132:40806) 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1ac97f5640 1 -- 172.21.15.132:0/2011527834 learned_addr learned my addr 172.21.15.132:0/2011527834 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.381+0000 7f1ac97f5640 1 -- 172.21.15.132:0/2011527834 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac0009450 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.382+0000 7f1ac97f5640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac4110b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006a70 tx=0x7f1ac002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.382+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac0031e90 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.382+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac003b910 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.382+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac41112a0 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.382+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac003a940 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.382+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac4111710 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.383+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1ac0039070 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.383+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac404b480 con 0x7f1ac4100cb0 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.383+0000 7f1ab9ffb640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1ab0034860 0x7f1ab0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:07.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.383+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1ac0066cf0 con 0x7f1ac4100cb0 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.383+0000 7f1ac8ff4640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1ab0034860 0x7f1ab0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.384+0000 7f1ac8ff4640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1ab0034860 0x7f1ab0036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4006fd0 tx=0x7f1ab4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.388+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1ac0042360 con 0x7f1ac4100cb0 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.587+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ac4111710 con 0x7f1ab0034860 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.592+0000 7f1ab9ffb640 1 -- 172.21.15.132:0/2011527834 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f1ac4111710 con 0x7f1ab0034860 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1ab0034860 msgr2=0x7f1ab0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1ab0034860 0x7f1ab0036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4006fd0 tx=0x7f1ab4006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 msgr2=0x7f1ac4110b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac4110b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006a70 tx=0x7f1ac002c040 comp rx=0 tx=0).stop 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 shutdown_connections 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f1ab0034860 0x7f1ab0036d10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 --2- 172.21.15.132:0/2011527834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ac4100cb0 0x7f1ac4110b60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:07.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 >> 172.21.15.132:0/2011527834 conn(0x7f1ac40fc490 msgr2=0x7f1ac40fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:07.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 shutdown_connections 2024-06-13T01:19:07.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:06.594+0000 7f1acba59640 1 -- 172.21.15.132:0/2011527834 wait complete. 2024-06-13T01:19:07.198 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying alertmanager service with default placement... 2024-06-13T01:19:07.713 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:07 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:07.713 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:07 smithi132 ceph-mon[29330]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:07.713 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:07 smithi132 ceph-mon[29330]: Saving service node-exporter spec with placement * 2024-06-13T01:19:07.713 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:07 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b641cb640 1 Processor -- start 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b641cb640 1 -- start start 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b641cb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c106d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b641cb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c102710 con 0x7f0b5c075bf0 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b61f67640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:08.512 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b61f67640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40812/0 (socket says 172.21.15.132:40812) 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.550+0000 7f0b61f67640 1 -- 172.21.15.132:0/1645380451 learned_addr learned my addr 172.21.15.132:0/1645380451 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.551+0000 7f0b61f67640 1 -- 172.21.15.132:0/1645380451 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c107260 con 0x7f0b5c075bf0 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.551+0000 7f0b61f67640 1 --2- 172.21.15.132:0/1645380451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c106d20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009840 tx=0x7f0b580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b0e8cb80451fee server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.551+0000 7f0b60f65640 1 -- 172.21.15.132:0/1645380451 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b58032440 con 0x7f0b5c075bf0 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b60f65640 1 -- 172.21.15.132:0/1645380451 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b58032a70 con 0x7f0b5c075bf0 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b60f65640 1 -- 172.21.15.132:0/1645380451 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b58031ad0 con 0x7f0b5c075bf0 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 -- 172.21.15.132:0/1645380451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 msgr2=0x7f0b5c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 --2- 172.21.15.132:0/1645380451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c106d20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009840 tx=0x7f0b580091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 -- 172.21.15.132:0/1645380451 shutdown_connections 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 --2- 172.21.15.132:0/1645380451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c106d20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 -- 172.21.15.132:0/1645380451 >> 172.21.15.132:0/1645380451 conn(0x7f0b5c0fc470 msgr2=0x7f0b5c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 -- 172.21.15.132:0/1645380451 shutdown_connections 2024-06-13T01:19:08.513 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.552+0000 7f0b641cb640 1 -- 172.21.15.132:0/1645380451 wait complete. 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b641cb640 1 Processor -- start 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b641cb640 1 -- start start 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b641cb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c112e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b641cb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c113360 con 0x7f0b5c075bf0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b61f67640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c112e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b61f67640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c112e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40828/0 (socket says 172.21.15.132:40828) 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b61f67640 1 -- 172.21.15.132:0/1169193434 learned_addr learned my addr 172.21.15.132:0/1169193434 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.553+0000 7f0b61f67640 1 -- 172.21.15.132:0/1169193434 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58009450 con 0x7f0b5c075bf0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.554+0000 7f0b61f67640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c112e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b580317f0 tx=0x7f0b5802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.554+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b58032880 con 0x7f0b5c075bf0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.554+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b5803b910 con 0x7f0b5c075bf0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.554+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c113560 con 0x7f0b5c075bf0 2024-06-13T01:19:08.514 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.554+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b5803a930 con 0x7f0b5c075bf0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.554+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c114040 con 0x7f0b5c075bf0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.555+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0b58039070 con 0x7f0b5c075bf0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.555+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c05ead0 con 0x7f0b5c075bf0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.555+0000 7f0b527fc640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f0b48034860 0x7f0b48036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.555+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0b58066dd0 con 0x7f0b5c075bf0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.555+0000 7f0b61766640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f0b48034860 0x7f0b48036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.556+0000 7f0b61766640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f0b48034860 0x7f0b48036d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c006fd0 tx=0x7f0b4c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.560+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0b5803ba80 con 0x7f0b5c075bf0 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.745+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b5c0fd870 con 0x7f0b48034860 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.750+0000 7f0b527fc640 1 -- 172.21.15.132:0/1169193434 <== mgr.14118 v2:172.21.15.132:6800/2668291289 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f0b5c0fd870 con 0x7f0b48034860 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f0b48034860 msgr2=0x7f0b48036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:08.515 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f0b48034860 0x7f0b48036d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c006fd0 tx=0x7f0b4c006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 msgr2=0x7f0b5c112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c112e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b580317f0 tx=0x7f0b5802c040 comp rx=0 tx=0).stop 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 shutdown_connections 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f0b48034860 0x7f0b48036d10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 --2- 172.21.15.132:0/1169193434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b5c075bf0 0x7f0b5c112e20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 >> 172.21.15.132:0/1169193434 conn(0x7f0b5c0fc470 msgr2=0x7f0b5c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 shutdown_connections 2024-06-13T01:19:08.516 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:07.752+0000 7f0b641cb640 1 -- 172.21.15.132:0/1169193434 wait complete. 2024-06-13T01:19:08.848 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:08 smithi132 ceph-mon[29330]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:08.849 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:08 smithi132 ceph-mon[29330]: Saving service alertmanager spec with placement count:1 2024-06-13T01:19:08.849 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:08 smithi132 ceph-mon[29330]: from='mgr.14118 172.21.15.132:0/2376202096' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:09.996 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:09 smithi132 ceph-mon[29330]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:09.996 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:09 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2653399881' entity='client.admin' 2024-06-13T01:19:10.316 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.930+0000 7efe046f4640 1 Processor -- start 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.930+0000 7efe046f4640 1 -- start start 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.930+0000 7efe046f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.930+0000 7efe046f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1026e0 con 0x7efdfc101d90 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.931+0000 7efe02490640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.931+0000 7efe02490640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc1021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40836/0 (socket says 172.21.15.132:40836) 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.931+0000 7efe02490640 1 -- 172.21.15.132:0/1729246136 learned_addr learned my addr 172.21.15.132:0/1729246136 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.931+0000 7efe02490640 1 -- 172.21.15.132:0/1729246136 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc102820 con 0x7efdfc101d90 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe02490640 1 --2- 172.21.15.132:0/1729246136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc1021a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009840 tx=0x7efdf80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5756d222ae9823 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe0148e640 1 -- 172.21.15.132:0/1729246136 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf8032440 con 0x7efdfc101d90 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe0148e640 1 -- 172.21.15.132:0/1729246136 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdf8032a70 con 0x7efdfc101d90 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe0148e640 1 -- 172.21.15.132:0/1729246136 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf8031ad0 con 0x7efdfc101d90 2024-06-13T01:19:10.317 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe046f4640 1 -- 172.21.15.132:0/1729246136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 msgr2=0x7efdfc1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe046f4640 1 --2- 172.21.15.132:0/1729246136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc1021a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009840 tx=0x7efdf80091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe046f4640 1 -- 172.21.15.132:0/1729246136 shutdown_connections 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe046f4640 1 --2- 172.21.15.132:0/1729246136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc1021a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe046f4640 1 -- 172.21.15.132:0/1729246136 >> 172.21.15.132:0/1729246136 conn(0x7efdfc0fd5b0 msgr2=0x7efdfc0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.932+0000 7efe046f4640 1 -- 172.21.15.132:0/1729246136 shutdown_connections 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe046f4640 1 -- 172.21.15.132:0/1729246136 wait complete. 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe046f4640 1 Processor -- start 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe046f4640 1 -- start start 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe046f4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc110ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe046f4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc111410 con 0x7efdfc101d90 2024-06-13T01:19:10.318 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe02490640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc110ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe02490640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc110ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40838/0 (socket says 172.21.15.132:40838) 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.933+0000 7efe02490640 1 -- 172.21.15.132:0/2653399881 learned_addr learned my addr 172.21.15.132:0/2653399881 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efe02490640 1 -- 172.21.15.132:0/2653399881 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8009450 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efe02490640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc110ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efdf8006a70 tx=0x7efdf802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf8031e90 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdf803b910 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc111610 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf803a940 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.934+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc112060 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.935+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7efdf8039070 con 0x7efdfc101d90 2024-06-13T01:19:10.319 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.935+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc05ead0 con 0x7efdfc101d90 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.935+0000 7efdf2ffd640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efde8034860 0x7efde8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.935+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efdf8066e10 con 0x7efdfc101d90 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.935+0000 7efe01c8f640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efde8034860 0x7efde8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.935+0000 7efe01c8f640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efde8034860 0x7efde8036d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efdec006fd0 tx=0x7efdec006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:08.939+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efdf8039320 con 0x7efdfc101d90 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.103+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7efdfc04b480 con 0x7efdfc101d90 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.106+0000 7efdf2ffd640 1 -- 172.21.15.132:0/2653399881 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7efdf803f070 con 0x7efdfc101d90 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.108+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efde8034860 msgr2=0x7efde8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.108+0000 7efe046f4640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efde8034860 0x7efde8036d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efdec006fd0 tx=0x7efdec006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.108+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 msgr2=0x7efdfc110ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.108+0000 7efe046f4640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc110ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efdf8006a70 tx=0x7efdf802c040 comp rx=0 tx=0).stop 2024-06-13T01:19:10.320 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.109+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 shutdown_connections 2024-06-13T01:19:10.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.109+0000 7efe046f4640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7efde8034860 0x7efde8036d10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:10.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.109+0000 7efe046f4640 1 --2- 172.21.15.132:0/2653399881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc101d90 0x7efdfc110ed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:10.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.109+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 >> 172.21.15.132:0/2653399881 conn(0x7efdfc0fd5b0 msgr2=0x7efdfc0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:10.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.109+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 shutdown_connections 2024-06-13T01:19:10.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:09.109+0000 7efe046f4640 1 -- 172.21.15.132:0/2653399881 wait complete. 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.759+0000 7f2c43486640 1 Processor -- start 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c43486640 1 -- start start 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c43486640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c43486640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c078160 con 0x7f2c3c077d50 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c41222640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c41222640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c076280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40844/0 (socket says 172.21.15.132:40844) 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c41222640 1 -- 172.21.15.132:0/992278982 learned_addr learned my addr 172.21.15.132:0/992278982 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.760+0000 7f2c41222640 1 -- 172.21.15.132:0/992278982 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c0767c0 con 0x7f2c3c077d50 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.761+0000 7f2c41222640 1 --2- 172.21.15.132:0/992278982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c076280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009840 tx=0x7f2c380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9fd4bb060fef83 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:11.521 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.761+0000 7f2c2ffff640 1 -- 172.21.15.132:0/992278982 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c38032440 con 0x7f2c3c077d50 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.761+0000 7f2c2ffff640 1 -- 172.21.15.132:0/992278982 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c38032a70 con 0x7f2c3c077d50 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.761+0000 7f2c2ffff640 1 -- 172.21.15.132:0/992278982 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c38031ad0 con 0x7f2c3c077d50 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 -- 172.21.15.132:0/992278982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 msgr2=0x7f2c3c076280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 --2- 172.21.15.132:0/992278982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c076280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009840 tx=0x7f2c380091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 -- 172.21.15.132:0/992278982 shutdown_connections 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 --2- 172.21.15.132:0/992278982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c076280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 -- 172.21.15.132:0/992278982 >> 172.21.15.132:0/992278982 conn(0x7f2c3c0fd490 msgr2=0x7f2c3c0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 -- 172.21.15.132:0/992278982 shutdown_connections 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 -- 172.21.15.132:0/992278982 wait complete. 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.762+0000 7f2c43486640 1 Processor -- start 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c43486640 1 -- start start 2024-06-13T01:19:11.522 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c43486640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c43486640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1136b0 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c41222640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c41222640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c113170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40848/0 (socket says 172.21.15.132:40848) 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c41222640 1 -- 172.21.15.132:0/838074676 learned_addr learned my addr 172.21.15.132:0/838074676 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c41222640 1 -- 172.21.15.132:0/838074676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38009450 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c41222640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c113170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098a0 tx=0x7f2c3802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c38031e10 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c3803b910 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c1138b0 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.763+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c3803a880 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.764+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c113d20 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.764+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f2c38039070 con 0x7f2c3c077d50 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.764+0000 7f2c2dffb640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f2c28034860 0x7f2c28036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:11.523 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.765+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2c38066c60 con 0x7f2c3c077d50 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.765+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c05ead0 con 0x7f2c3c077d50 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.765+0000 7f2c40a21640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f2c28034860 0x7f2c28036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.765+0000 7f2c40a21640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f2c28034860 0x7f2c28036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c30006fd0 tx=0x7f2c30006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.769+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2c38037070 con 0x7f2c3c077d50 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.932+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f2c3c108120 con 0x7f2c3c077d50 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.935+0000 7f2c2dffb640 1 -- 172.21.15.132:0/838074676 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f2c3803acb0 con 0x7f2c3c077d50 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f2c28034860 msgr2=0x7f2c28036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f2c28034860 0x7f2c28036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c30006fd0 tx=0x7f2c30006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 msgr2=0x7f2c3c113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c113170 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098a0 tx=0x7f2c3802c040 comp rx=0 tx=0).stop 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 shutdown_connections 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f2c28034860 0x7f2c28036d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:11.524 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 --2- 172.21.15.132:0/838074676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c3c077d50 0x7f2c3c113170 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:11.525 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 >> 172.21.15.132:0/838074676 conn(0x7f2c3c0fd490 msgr2=0x7f2c3c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:11.525 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 shutdown_connections 2024-06-13T01:19:11.525 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:10.937+0000 7f2c43486640 1 -- 172.21.15.132:0/838074676 wait complete. 2024-06-13T01:19:11.525 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling the dashboard module... 2024-06-13T01:19:12.085 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:11 smithi132 ceph-mon[29330]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:12.085 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:11 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/838074676' entity='client.admin' 2024-06-13T01:19:13.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:12 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1559929050' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-13T01:19:13.586 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.869+0000 7f08477a4640 1 Processor -- start 2024-06-13T01:19:13.586 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.869+0000 7f08477a4640 1 -- start start 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.870+0000 7f08477a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840102120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.870+0000 7f08477a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840102660 con 0x7f0840101d10 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.870+0000 7f0845540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.870+0000 7f0845540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40854/0 (socket says 172.21.15.132:40854) 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.870+0000 7f0845540640 1 -- 172.21.15.132:0/416220682 learned_addr learned my addr 172.21.15.132:0/416220682 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.870+0000 7f0845540640 1 -- 172.21.15.132:0/416220682 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08401027a0 con 0x7f0840101d10 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.871+0000 7f0845540640 1 --2- 172.21.15.132:0/416220682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840102120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f083c009840 tx=0x7f083c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754dfebf51e9626d server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.871+0000 7f0833fff640 1 -- 172.21.15.132:0/416220682 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c032440 con 0x7f0840101d10 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.871+0000 7f0833fff640 1 -- 172.21.15.132:0/416220682 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f083c032a70 con 0x7f0840101d10 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.871+0000 7f0833fff640 1 -- 172.21.15.132:0/416220682 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c031ad0 con 0x7f0840101d10 2024-06-13T01:19:13.587 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.871+0000 7f08477a4640 1 -- 172.21.15.132:0/416220682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 msgr2=0x7f0840102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.871+0000 7f08477a4640 1 --2- 172.21.15.132:0/416220682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840102120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f083c009840 tx=0x7f083c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 -- 172.21.15.132:0/416220682 shutdown_connections 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 --2- 172.21.15.132:0/416220682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840102120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 -- 172.21.15.132:0/416220682 >> 172.21.15.132:0/416220682 conn(0x7f08400fd4f0 msgr2=0x7f08400ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 -- 172.21.15.132:0/416220682 shutdown_connections 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 -- 172.21.15.132:0/416220682 wait complete. 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 Processor -- start 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 -- start start 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840110f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f08477a4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840111440 con 0x7f0840101d10 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f0845540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840110f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f0845540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840110f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40858/0 (socket says 172.21.15.132:40858) 2024-06-13T01:19:13.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.872+0000 7f0845540640 1 -- 172.21.15.132:0/1559929050 learned_addr learned my addr 172.21.15.132:0/1559929050 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f0845540640 1 -- 172.21.15.132:0/1559929050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c009450 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f0845540640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840110f00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f083c006a70 tx=0x7f083c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c031e90 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f083c03b910 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c03a940 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0840111640 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.873+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0840112090 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.874+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f083c039070 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.874+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084005ead0 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.874+0000 7f0831ffb640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f082c034860 0x7f082c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.874+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f083c066e10 con 0x7f0840101d10 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.874+0000 7f0844d3f640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f082c034860 0x7f082c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:13.589 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.875+0000 7f0844d3f640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f082c034860 0x7f082c036d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0834006fd0 tx=0x7f0834006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:11.879+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f083c039500 con 0x7f0840101d10 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.087+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0840103d00 con 0x7f0840101d10 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.941+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f083c03b580 con 0x7f0840101d10 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.943+0000 7f0831ffb640 1 -- 172.21.15.132:0/1559929050 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f083c0433b0 con 0x7f0840101d10 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.945+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f082c034860 msgr2=0x7f082c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f082c034860 0x7f082c036d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0834006fd0 tx=0x7f0834006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 msgr2=0x7f0840110f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840110f00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f083c006a70 tx=0x7f083c02c040 comp rx=0 tx=0).stop 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 shutdown_connections 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f082c034860 0x7f082c036d10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 --2- 172.21.15.132:0/1559929050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0840101d10 0x7f0840110f00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 >> 172.21.15.132:0/1559929050 conn(0x7f08400fd4f0 msgr2=0x7f08400fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:13.590 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 shutdown_connections 2024-06-13T01:19:13.591 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:12.946+0000 7f08477a4640 1 -- 172.21.15.132:0/1559929050 wait complete. 2024-06-13T01:19:14.180 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:13 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1559929050' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-13T01:19:14.180 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:13 smithi132 ceph-mon[29330]: mgrmap e9: smithi132.vjweoa(active, since 24s) 2024-06-13T01:19:14.877 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:19:14.877 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "active_name": "smithi132.vjweoa", 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f62a04b5640 1 Processor -- start 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f62a04b5640 1 -- start start 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f62a04b5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f62a04b5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298077fe0 con 0x7f6298077bd0 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f629e251640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f629e251640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55366/0 (socket says 172.21.15.132:55366) 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.976+0000 7f629e251640 1 -- 172.21.15.132:0/2043846611 learned_addr learned my addr 172.21.15.132:0/2043846611 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:14.878 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.977+0000 7f629e251640 1 -- 172.21.15.132:0/2043846611 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6298078120 con 0x7f6298077bd0 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.977+0000 7f629e251640 1 --2- 172.21.15.132:0/2043846611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298076240 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6294009840 tx=0x7f62940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fde58769d8bf0bf server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.977+0000 7f629d24f640 1 -- 172.21.15.132:0/2043846611 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6294032440 con 0x7f6298077bd0 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.977+0000 7f629d24f640 1 -- 172.21.15.132:0/2043846611 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6294032a70 con 0x7f6298077bd0 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.977+0000 7f629d24f640 1 -- 172.21.15.132:0/2043846611 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6294031ad0 con 0x7f6298077bd0 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.978+0000 7f62a04b5640 1 -- 172.21.15.132:0/2043846611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 msgr2=0x7f6298076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.978+0000 7f62a04b5640 1 --2- 172.21.15.132:0/2043846611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298076240 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6294009840 tx=0x7f62940091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.978+0000 7f62a04b5640 1 -- 172.21.15.132:0/2043846611 shutdown_connections 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.978+0000 7f62a04b5640 1 --2- 172.21.15.132:0/2043846611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298076240 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 -- 172.21.15.132:0/2043846611 >> 172.21.15.132:0/2043846611 conn(0x7f62980fd5b0 msgr2=0x7f62980ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 -- 172.21.15.132:0/2043846611 shutdown_connections 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 -- 172.21.15.132:0/2043846611 wait complete. 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 Processor -- start 2024-06-13T01:19:14.879 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 -- start start 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298112e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.979+0000 7f62a04b5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298113370 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f629e251640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298112e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f629e251640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298112e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55376/0 (socket says 172.21.15.132:55376) 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f629e251640 1 -- 172.21.15.132:0/1048508495 learned_addr learned my addr 172.21.15.132:0/1048508495 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f629e251640 1 -- 172.21.15.132:0/1048508495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294009450 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f629e251640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298112e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f62940317f0 tx=0x7f629402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6294032900 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6298113570 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f629403b910 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629403a960 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.980+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6298113fc0 con 0x7f6298077bd0 2024-06-13T01:19:14.880 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.981+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f6294039070 con 0x7f6298077bd0 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.981+0000 7f628effd640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 0x7f6284036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.981+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f629403b440 con 0x7f6298077bd0 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.981+0000 7f629da50640 1 -- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 msgr2=0x7f6284036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/2668291289 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.981+0000 7f629da50640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 0x7f6284036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.981+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629805ead0 con 0x7f6298077bd0 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:13.990+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6294037020 con 0x7f6298077bd0 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.182+0000 7f629da50640 1 -- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 msgr2=0x7f6284036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/2668291289 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.182+0000 7f629da50640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 0x7f6284036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.225+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f6298107f70 con 0x7f6298077bd0 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.226+0000 7f628effd640 1 -- 172.21.15.132:0/1048508495 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f629403f070 con 0x7f6298077bd0 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 msgr2=0x7f6284036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:19:14.881 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 0x7f6284036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 msgr2=0x7f6298112e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298112e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f62940317f0 tx=0x7f629402c040 comp rx=0 tx=0).stop 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 shutdown_connections 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7f6284034860 0x7f6284036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 --2- 172.21.15.132:0/1048508495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6298077bd0 0x7f6298112e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 >> 172.21.15.132:0/1048508495 conn(0x7f62980fd5b0 msgr2=0x7f62980fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 shutdown_connections 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:14.228+0000 7f62a04b5640 1 -- 172.21.15.132:0/1048508495 wait complete. 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for the mgr to restart... 2024-06-13T01:19:14.882 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr epoch 9... 2024-06-13T01:19:15.136 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:14 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1048508495' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: Active manager daemon smithi132.vjweoa restarted 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: Activating manager daemon smithi132.vjweoa 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: osdmap e3: 0 total, 0 up, 0 in 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: mgrmap e10: smithi132.vjweoa(active, starting, since 0.0031605s) 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr metadata", "who": "smithi132.vjweoa", "id": "smithi132.vjweoa"}]: dispatch 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T01:19:17.650 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T01:19:17.651 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: Manager daemon smithi132.vjweoa is now available 2024-06-13T01:19:17.651 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:19:17.651 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:19:17.651 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/mirror_snapshot_schedule"}]: dispatch 2024-06-13T01:19:17.651 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:17 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/trash_purge_schedule"}]: dispatch 2024-06-13T01:19:19.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-06-13T01:19:19.321 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.292+0000 7fcdeb8cb640 1 Processor -- start 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.292+0000 7fcdeb8cb640 1 -- start start 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.292+0000 7fcdeb8cb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4106e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.292+0000 7fcdeb8cb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4075ee0 con 0x7fcde4075ad0 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.293+0000 7fcde9667640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.293+0000 7fcde9667640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55378/0 (socket says 172.21.15.132:55378) 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.293+0000 7fcde9667640 1 -- 172.21.15.132:0/2162759072 learned_addr learned my addr 172.21.15.132:0/2162759072 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.293+0000 7fcde9667640 1 -- 172.21.15.132:0/2162759072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4102810 con 0x7fcde4075ad0 2024-06-13T01:19:19.322 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.294+0000 7fcde9667640 1 --2- 172.21.15.132:0/2162759072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4106e20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009840 tx=0x7fcde00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792434568b412902 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.294+0000 7fcddbfff640 1 -- 172.21.15.132:0/2162759072 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde0032440 con 0x7fcde4075ad0 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.294+0000 7fcddbfff640 1 -- 172.21.15.132:0/2162759072 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcde0032a70 con 0x7fcde4075ad0 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.294+0000 7fcddbfff640 1 -- 172.21.15.132:0/2162759072 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde0031ad0 con 0x7fcde4075ad0 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.294+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/2162759072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 msgr2=0x7fcde4106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.294+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/2162759072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4106e20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009840 tx=0x7fcde00091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/2162759072 shutdown_connections 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/2162759072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4106e20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/2162759072 >> 172.21.15.132:0/2162759072 conn(0x7fcde40fc490 msgr2=0x7fcde40fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/2162759072 shutdown_connections 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/2162759072 wait complete. 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 Processor -- start 2024-06-13T01:19:19.323 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.295+0000 7fcdeb8cb640 1 -- start start 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcdeb8cb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4112e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcdeb8cb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4113340 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcde9667640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4112e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcde9667640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4112e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55384/0 (socket says 172.21.15.132:55384) 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcde9667640 1 -- 172.21.15.132:0/1922339601 learned_addr learned my addr 172.21.15.132:0/1922339601 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcde9667640 1 -- 172.21.15.132:0/1922339601 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde0009450 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcde9667640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4112e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcde00317f0 tx=0x7fcde002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde0032900 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcde003b910 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde41135a0 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.296+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde003a960 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.297+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde4113f90 con 0x7fcde4075ad0 2024-06-13T01:19:19.324 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.298+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fcde0039070 con 0x7fcde4075ad0 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.298+0000 7fcdd9ffb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.298+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 --> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcde0032900 con 0x7fcdd00348b0 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.298+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcde0066e80 con 0x7fcde4075ad0 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.298+0000 7fcde8e66640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 msgr2=0x7fcdd0036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/2668291289 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.298+0000 7fcde8e66640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.499+0000 7fcde8e66640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 msgr2=0x7fcdd0036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/2668291289 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.499+0000 7fcde8e66640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.900+0000 7fcde8e66640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 msgr2=0x7fcdd0036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/2668291289 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:15.900+0000 7fcde8e66640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:16.701+0000 7fcde8e66640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 msgr2=0x7fcdd0036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/2668291289 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:16.701+0000 7fcde8e66640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-13T01:19:19.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:17.500+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7fcde0066020 con 0x7fcde4075ad0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:17.500+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 msgr2=0x7fcdd0036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:17.500+0000 7fcdd9ffb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.501+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7fcde004c350 con 0x7fcde4075ad0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.502+0000 7fcdd9ffb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fcdd0037fd0 0x7fcdd003a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.502+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcde0032900 con 0x7fcdd0037fd0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.502+0000 7fcde8e66640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fcdd0037fd0 0x7fcdd003a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.502+0000 7fcde8e66640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fcdd0037fd0 0x7fcdd003a3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4003940 tx=0x7fcdd4009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.503+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fcde0032900 con 0x7fcdd0037fd0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.508+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcde405ead0 con 0x7fcdd0037fd0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.508+0000 7fcdd9ffb640 1 -- 172.21.15.132:0/1922339601 <== mgr.14160 v2:172.21.15.132:6800/3829496564 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fcde405ead0 con 0x7fcdd0037fd0 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fcdd0037fd0 msgr2=0x7fcdd003a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:19.326 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fcdd0037fd0 0x7fcdd003a3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4003940 tx=0x7fcdd4009350 comp rx=0 tx=0).stop 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 msgr2=0x7fcde4112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4112e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcde00317f0 tx=0x7fcde002c040 comp rx=0 tx=0).stop 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 shutdown_connections 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fcdd0037fd0 0x7fcdd003a3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:6800/2668291289,v1:172.21.15.132:6801/2668291289] conn(0x7fcdd00348b0 0x7fcdd0036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 --2- 172.21.15.132:0/1922339601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcde4075ad0 0x7fcde4112e00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 >> 172.21.15.132:0/1922339601 conn(0x7fcde40fc490 msgr2=0x7fcde40fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 shutdown_connections 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:18.509+0000 7fcdeb8cb640 1 -- 172.21.15.132:0/1922339601 wait complete. 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:mgr epoch 9 is available 2024-06-13T01:19:19.327 INFO:teuthology.orchestra.run.smithi132.stdout:Generating a dashboard self-signed certificate... 2024-06-13T01:19:19.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: mgrmap e11: smithi132.vjweoa(active, since 1.00557s) 2024-06-13T01:19:19.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: [13/Jun/2024:01:19:18] ENGINE Bus STARTING 2024-06-13T01:19:19.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: [13/Jun/2024:01:19:18] ENGINE Serving on https://172.21.15.132:7150 2024-06-13T01:19:19.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: [13/Jun/2024:01:19:18] ENGINE Bus STARTED 2024-06-13T01:19:19.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: [13/Jun/2024:01:19:18] ENGINE Error in HTTPServer.serve 2024-06-13T01:19:19.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Traceback (most recent call last): 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._connections.run(self.expiration_interval) 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._run(expiration_interval) 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: s = self.context.wrap_socket( 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: return self.sslsocket_class._create( 2024-06-13T01:19:19.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T01:19:19.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self.do_handshake() 2024-06-13T01:19:19.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T01:19:19.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._sslobj.do_handshake() 2024-06-13T01:19:19.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T01:19:19.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:19.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:19 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.752+0000 7f7b5b6c9640 1 Processor -- start 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.752+0000 7f7b5b6c9640 1 -- start start 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.753+0000 7f7b5b6c9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b54101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.753+0000 7f7b5b6c9640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b541015a0 con 0x7f7b54100c50 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.753+0000 7f7b59465640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b54101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.753+0000 7f7b59465640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b54101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55448/0 (socket says 172.21.15.132:55448) 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.753+0000 7f7b59465640 1 -- 172.21.15.132:0/210074743 learned_addr learned my addr 172.21.15.132:0/210074743 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.753+0000 7f7b59465640 1 -- 172.21.15.132:0/210074743 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b541016e0 con 0x7f7b54100c50 2024-06-13T01:19:20.602 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b59465640 1 --2- 172.21.15.132:0/210074743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b54101060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009840 tx=0x7f7b500091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccf3ee1c118f441 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b4bfff640 1 -- 172.21.15.132:0/210074743 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b50032440 con 0x7f7b54100c50 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b4bfff640 1 -- 172.21.15.132:0/210074743 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b50032a70 con 0x7f7b54100c50 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b4bfff640 1 -- 172.21.15.132:0/210074743 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b50031ad0 con 0x7f7b54100c50 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/210074743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 msgr2=0x7f7b54101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b5b6c9640 1 --2- 172.21.15.132:0/210074743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b54101060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009840 tx=0x7f7b500091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/210074743 shutdown_connections 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b5b6c9640 1 --2- 172.21.15.132:0/210074743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b54101060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.754+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/210074743 >> 172.21.15.132:0/210074743 conn(0x7f7b540fc430 msgr2=0x7f7b540fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/210074743 shutdown_connections 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/210074743 wait complete. 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b5b6c9640 1 Processor -- start 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b5b6c9640 1 -- start start 2024-06-13T01:19:20.603 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b5b6c9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b5410fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b5b6c9640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54110010 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b59465640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b5410fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b59465640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b5410fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55454/0 (socket says 172.21.15.132:55454) 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.755+0000 7f7b59465640 1 -- 172.21.15.132:0/769536075 learned_addr learned my addr 172.21.15.132:0/769536075 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b59465640 1 -- 172.21.15.132:0/769536075 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50009450 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b59465640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b5410fad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009810 tx=0x7f7b5002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b500328b0 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b5003b910 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b54110210 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b5003a980 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.756+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b54110680 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.757+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f7b50039070 con 0x7f7b54100c50 2024-06-13T01:19:20.604 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.757+0000 7f7b49ffb640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7b40034730 0x7f7b40036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.757+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7b50066be0 con 0x7f7b54100c50 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.757+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b54110680 con 0x7f7b54100c50 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.757+0000 7f7b58c64640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7b40034730 0x7f7b40036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.758+0000 7f7b58c64640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7b40034730 0x7f7b40036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b44006fd0 tx=0x7f7b44006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.762+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7b54110680 con 0x7f7b54100c50 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.956+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b54110680 con 0x7f7b40034730 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:19.998+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f7b50040260 con 0x7f7b54100c50 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.022+0000 7f7b49ffb640 1 -- 172.21.15.132:0/769536075 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f7b54110680 con 0x7f7b40034730 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7b40034730 msgr2=0x7f7b40036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7b40034730 0x7f7b40036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b44006fd0 tx=0x7f7b44006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 msgr2=0x7f7b5410fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:20.605 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b5410fad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009810 tx=0x7f7b5002c040 comp rx=0 tx=0).stop 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 shutdown_connections 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7b40034730 0x7f7b40036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 --2- 172.21.15.132:0/769536075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b54100c50 0x7f7b5410fad0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.025+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 >> 172.21.15.132:0/769536075 conn(0x7f7b540fc430 msgr2=0x7f7b540fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.026+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 shutdown_connections 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.026+0000 7f7b5b6c9640 1 -- 172.21.15.132:0/769536075 wait complete. 2024-06-13T01:19:20.606 INFO:teuthology.orchestra.run.smithi132.stdout:Creating initial admin user... 2024-06-13T01:19:20.981 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:20 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:20.981 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:20 smithi132 ceph-mon[29330]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:20.981 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:20 smithi132 ceph-mon[29330]: mgrmap e12: smithi132.vjweoa(active, since 2s) 2024-06-13T01:19:20.981 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:20 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:20.981 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:20 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:22.196 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$RCdwObyJ.qwH/JIea7LLsOGl6FIWGKO4n314hFJLXExw2.iIWOvTy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718241561, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68caad9640 1 Processor -- start 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68caad9640 1 -- start start 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68caad9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4104a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68caad9640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4075f00 con 0x7f68c4075af0 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68c8875640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68c8875640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4104a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55462/0 (socket says 172.21.15.132:55462) 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68c8875640 1 -- 172.21.15.132:0/1436784135 learned_addr learned my addr 172.21.15.132:0/1436784135 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.987+0000 7f68c8875640 1 -- 172.21.15.132:0/1436784135 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c4104f40 con 0x7f68c4075af0 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.988+0000 7f68c8875640 1 --2- 172.21.15.132:0/1436784135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4104a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009840 tx=0x7f68b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4bdd19d4160e7f server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.988+0000 7f68c37fe640 1 -- 172.21.15.132:0/1436784135 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8032440 con 0x7f68c4075af0 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.988+0000 7f68c37fe640 1 -- 172.21.15.132:0/1436784135 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b8032a70 con 0x7f68c4075af0 2024-06-13T01:19:22.197 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.988+0000 7f68c37fe640 1 -- 172.21.15.132:0/1436784135 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8031ad0 con 0x7f68c4075af0 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 -- 172.21.15.132:0/1436784135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 msgr2=0x7f68c4104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 --2- 172.21.15.132:0/1436784135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4104a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009840 tx=0x7f68b80091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 -- 172.21.15.132:0/1436784135 shutdown_connections 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 --2- 172.21.15.132:0/1436784135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4104a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 -- 172.21.15.132:0/1436784135 >> 172.21.15.132:0/1436784135 conn(0x7f68c40fc430 msgr2=0x7f68c40fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 -- 172.21.15.132:0/1436784135 shutdown_connections 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 -- 172.21.15.132:0/1436784135 wait complete. 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.989+0000 7f68caad9640 1 Processor -- start 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68caad9640 1 -- start start 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68caad9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4110bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68caad9640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4111100 con 0x7f68c4075af0 2024-06-13T01:19:22.198 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68c8875640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4110bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68c8875640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4110bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55466/0 (socket says 172.21.15.132:55466) 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68c8875640 1 -- 172.21.15.132:0/1597064835 learned_addr learned my addr 172.21.15.132:0/1597064835 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68c8875640 1 -- 172.21.15.132:0/1597064835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8009450 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68c8875640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4110bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68b80317f0 tx=0x7f68b802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.990+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8032900 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.991+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c4111300 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.991+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b803b910 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.991+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b803a960 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.991+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c4111770 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.991+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f68b8039070 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.992+0000 7f68c17fa640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f68b4034860 0x7f68b4036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.992+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f68b8066fb0 con 0x7f68c4075af0 2024-06-13T01:19:22.199 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.992+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c405ead0 con 0x7f68c4075af0 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.992+0000 7f68c3fff640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f68b4034860 0x7f68b4036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.992+0000 7f68c3fff640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f68b4034860 0x7f68b4036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68b0006fd0 tx=0x7f68b0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:20.996+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f68b8037070 con 0x7f68c4075af0 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.184+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f68c404b480 con 0x7f68b4034860 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.410+0000 7f68c17fa640 1 -- 172.21.15.132:0/1597064835 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f68c404b480 con 0x7f68b4034860 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f68b4034860 msgr2=0x7f68b4036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f68b4034860 0x7f68b4036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68b0006fd0 tx=0x7f68b0006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 msgr2=0x7f68c4110bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4110bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68b80317f0 tx=0x7f68b802c040 comp rx=0 tx=0).stop 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 shutdown_connections 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f68b4034860 0x7f68b4036d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:22.200 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 --2- 172.21.15.132:0/1597064835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68c4075af0 0x7f68c4110bc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:22.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.412+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 >> 172.21.15.132:0/1597064835 conn(0x7f68c40fc430 msgr2=0x7f68c40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:22.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.413+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 shutdown_connections 2024-06-13T01:19:22.201 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:21.413+0000 7f68caad9640 1 -- 172.21.15.132:0/1597064835 wait complete. 2024-06-13T01:19:22.201 INFO:teuthology.orchestra.run.smithi132.stdout:Fetching dashboard port number... 2024-06-13T01:19:22.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:22 smithi132 ceph-mon[29330]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:22.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:22 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:23.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:23 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1530205224' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 8443 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f641a21a640 1 Processor -- start 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f641a21a640 1 -- start start 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f641a21a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f64141021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f641a21a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141026e0 con 0x7f6414101d90 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f64137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f64141021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:23.830 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f64137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f64141021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34790/0 (socket says 172.21.15.132:34790) 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f64137fe640 1 -- 172.21.15.132:0/1321439475 learned_addr learned my addr 172.21.15.132:0/1321439475 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.055+0000 7f64137fe640 1 -- 172.21.15.132:0/1321439475 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414102820 con 0x7f6414101d90 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.056+0000 7f64137fe640 1 --2- 172.21.15.132:0/1321439475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f64141021a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6404009840 tx=0x7f64040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af699badbf3389c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.056+0000 7f64127fc640 1 -- 172.21.15.132:0/1321439475 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6404032440 con 0x7f6414101d90 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.056+0000 7f64127fc640 1 -- 172.21.15.132:0/1321439475 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6404032a70 con 0x7f6414101d90 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.056+0000 7f64127fc640 1 -- 172.21.15.132:0/1321439475 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6404031ad0 con 0x7f6414101d90 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 -- 172.21.15.132:0/1321439475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 msgr2=0x7f64141021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 --2- 172.21.15.132:0/1321439475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f64141021a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6404009840 tx=0x7f64040091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 -- 172.21.15.132:0/1321439475 shutdown_connections 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 --2- 172.21.15.132:0/1321439475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f64141021a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:23.831 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 -- 172.21.15.132:0/1321439475 >> 172.21.15.132:0/1321439475 conn(0x7f64140fd590 msgr2=0x7f64140ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 -- 172.21.15.132:0/1321439475 shutdown_connections 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 -- 172.21.15.132:0/1321439475 wait complete. 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.057+0000 7f641a21a640 1 Processor -- start 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f641a21a640 1 -- start start 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f641a21a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f6414110b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f641a21a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141110b0 con 0x7f6414101d90 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f64137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f6414110b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f64137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f6414110b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34792/0 (socket says 172.21.15.132:34792) 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f64137fe640 1 -- 172.21.15.132:0/1530205224 learned_addr learned my addr 172.21.15.132:0/1530205224 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f64137fe640 1 -- 172.21.15.132:0/1530205224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6404009450 con 0x7f6414101d90 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f64137fe640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f6414110b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6404009810 tx=0x7f640402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.058+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64040328b0 con 0x7f6414101d90 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64141112b0 con 0x7f6414101d90 2024-06-13T01:19:23.832 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f640403b910 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f640403a980 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6414111720 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f6404039070 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f63fbfff640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f63fc034860 0x7f63fc036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.059+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6404066e10 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.060+0000 7f6412ffd640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f63fc034860 0x7f63fc036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.060+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641405ead0 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.060+0000 7f6412ffd640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f63fc034860 0x7f63fc036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6408006fd0 tx=0x7f6408006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.064+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f640403f020 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.228+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f641404b480 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.228+0000 7f63fbfff640 1 -- 172.21.15.132:0/1530205224 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f6404037030 con 0x7f6414101d90 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f63fc034860 msgr2=0x7f63fc036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f63fc034860 0x7f63fc036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6408006fd0 tx=0x7f6408006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:23.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 msgr2=0x7f6414110b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f6414110b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6404009810 tx=0x7f640402c040 comp rx=0 tx=0).stop 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 shutdown_connections 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f63fc034860 0x7f63fc036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 --2- 172.21.15.132:0/1530205224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6414101d90 0x7f6414110b70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.230+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 >> 172.21.15.132:0/1530205224 conn(0x7f64140fd590 msgr2=0x7f64140fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.231+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 shutdown_connections 2024-06-13T01:19:23.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:23.231+0000 7f641a21a640 1 -- 172.21.15.132:0/1530205224 wait complete. 2024-06-13T01:19:23.837 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-13T01:19:23.837 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout disabled 2024-06-13T01:19:23.845 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-13T01:19:23.845 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-06-13T01:19:23.846 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld.service is not enabled 2024-06-13T01:19:23.846 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout:Ceph Dashboard is now available at: 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout: URL: https://smithi132.front.sepia.ceph.com:8443/ 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout: User: admin 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout: Password: wpmp7od4o8 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout:Saving cluster configuration to /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config directory 2024-06-13T01:19:23.848 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling autotune for osd_memory_target 2024-06-13T01:19:25.070 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.289+0000 7f5a7e6f6640 1 Processor -- start 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.289+0000 7f5a7e6f6640 1 -- start start 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.289+0000 7f5a7e6f6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 0x7f5a78102160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.289+0000 7f5a7e6f6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a781026a0 con 0x7f5a78101d50 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.290+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 0x7f5a78102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.290+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 0x7f5a78102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34806/0 (socket says 172.21.15.132:34806) 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.290+0000 7f5a77fff640 1 -- 172.21.15.132:0/4054798394 learned_addr learned my addr 172.21.15.132:0/4054798394 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.290+0000 7f5a77fff640 1 -- 172.21.15.132:0/4054798394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a781027e0 con 0x7f5a78101d50 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.290+0000 7f5a77fff640 1 --2- 172.21.15.132:0/4054798394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 0x7f5a78102160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009840 tx=0x7f5a680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150d4acfe320675f server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a76ffd640 1 -- 172.21.15.132:0/4054798394 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a68032440 con 0x7f5a78101d50 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a76ffd640 1 -- 172.21.15.132:0/4054798394 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a68032a70 con 0x7f5a78101d50 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a76ffd640 1 -- 172.21.15.132:0/4054798394 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a68031ad0 con 0x7f5a78101d50 2024-06-13T01:19:25.071 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/4054798394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 msgr2=0x7f5a78102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 --2- 172.21.15.132:0/4054798394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 0x7f5a78102160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009840 tx=0x7f5a680091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/4054798394 shutdown_connections 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 --2- 172.21.15.132:0/4054798394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78101d50 0x7f5a78102160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/4054798394 >> 172.21.15.132:0/4054798394 conn(0x7f5a780fd590 msgr2=0x7f5a780ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/4054798394 shutdown_connections 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.291+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/4054798394 wait complete. 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a7e6f6640 1 Processor -- start 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a7e6f6640 1 -- start start 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a7e6f6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 0x7f5a78111370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a7e6f6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a781118b0 con 0x7f5a78110f60 2024-06-13T01:19:25.072 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 0x7f5a78111370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 0x7f5a78111370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34812/0 (socket says 172.21.15.132:34812) 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a77fff640 1 -- 172.21.15.132:0/3532687979 learned_addr learned my addr 172.21.15.132:0/3532687979 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.292+0000 7f5a77fff640 1 -- 172.21.15.132:0/3532687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a68009450 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.293+0000 7f5a77fff640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 0x7f5a78111370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5a680098a0 tx=0x7f5a680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.293+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a680327b0 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.293+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a6803d910 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.293+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a78111ab0 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.293+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a6803b7e0 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.293+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a78112490 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.294+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f5a68039040 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.294+0000 7f5a74ff9640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5a600348b0 0x7f5a60036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.294+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5a68067510 con 0x7f5a78110f60 2024-06-13T01:19:25.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.294+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7805ead0 con 0x7f5a78110f60 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.294+0000 7f5a777fe640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5a600348b0 0x7f5a60036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.294+0000 7f5a777fe640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5a600348b0 0x7f5a60036d60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006fd0 tx=0x7f5a6c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.298+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5a68041080 con 0x7f5a78110f60 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.412+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f5a6803dad0 con 0x7f5a78110f60 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.465+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5a7804b480 con 0x7f5a78110f60 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.465+0000 7f5a74ff9640 1 -- 172.21.15.132:0/3532687979 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5a68037070 con 0x7f5a78110f60 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5a600348b0 msgr2=0x7f5a60036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5a600348b0 0x7f5a60036d60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006fd0 tx=0x7f5a6c006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 msgr2=0x7f5a78111370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 0x7f5a78111370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5a680098a0 tx=0x7f5a680091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 shutdown_connections 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5a600348b0 0x7f5a60036d60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:25.074 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 --2- 172.21.15.132:0/3532687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a78110f60 0x7f5a78111370 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:25.075 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 >> 172.21.15.132:0/3532687979 conn(0x7f5a780fd590 msgr2=0x7f5a780fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:25.075 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 shutdown_connections 2024-06-13T01:19:25.075 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:24.467+0000 7f5a7e6f6640 1 -- 172.21.15.132:0/3532687979 wait complete. 2024-06-13T01:19:25.691 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:25 smithi132 ceph-mon[29330]: mgrmap e13: smithi132.vjweoa(active, since 6s) 2024-06-13T01:19:26.276 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a81bb1640 1 Processor -- start 2024-06-13T01:19:26.276 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a81bb1640 1 -- start start 2024-06-13T01:19:26.276 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a81bb1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c103770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:26.276 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a81bb1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c0708a0 con 0x7f4a7c103360 2024-06-13T01:19:26.276 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a7b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a7b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34814/0 (socket says 172.21.15.132:34814) 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.432+0000 7f4a7b7fe640 1 -- 172.21.15.132:0/2084850800 learned_addr learned my addr 172.21.15.132:0/2084850800 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.433+0000 7f4a7b7fe640 1 -- 172.21.15.132:0/2084850800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c103cb0 con 0x7f4a7c103360 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.433+0000 7f4a7b7fe640 1 --2- 172.21.15.132:0/2084850800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c103770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009840 tx=0x7f4a6c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7191052a09ba7f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a7affd640 1 -- 172.21.15.132:0/2084850800 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c032440 con 0x7f4a7c103360 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a7affd640 1 -- 172.21.15.132:0/2084850800 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a6c032a70 con 0x7f4a7c103360 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a7affd640 1 -- 172.21.15.132:0/2084850800 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c031ad0 con 0x7f4a7c103360 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 -- 172.21.15.132:0/2084850800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 msgr2=0x7f4a7c103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 --2- 172.21.15.132:0/2084850800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c103770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009840 tx=0x7f4a6c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 -- 172.21.15.132:0/2084850800 shutdown_connections 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 --2- 172.21.15.132:0/2084850800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c103770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 -- 172.21.15.132:0/2084850800 >> 172.21.15.132:0/2084850800 conn(0x7f4a7c0fbc20 msgr2=0x7f4a7c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 -- 172.21.15.132:0/2084850800 shutdown_connections 2024-06-13T01:19:26.277 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.434+0000 7f4a81bb1640 1 -- 172.21.15.132:0/2084850800 wait complete. 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a81bb1640 1 Processor -- start 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a81bb1640 1 -- start start 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a81bb1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c110b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a81bb1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1110d0 con 0x7f4a7c103360 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a7b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c110b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a7b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c110b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34822/0 (socket says 172.21.15.132:34822) 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.435+0000 7f4a7b7fe640 1 -- 172.21.15.132:0/3013306685 learned_addr learned my addr 172.21.15.132:0/3013306685 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a7b7fe640 1 -- 172.21.15.132:0/3013306685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c009450 con 0x7f4a7c103360 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a7b7fe640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c110b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009810 tx=0x7f4a6c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c0328b0 con 0x7f4a7c103360 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a6c03b910 con 0x7f4a7c103360 2024-06-13T01:19:26.278 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c1112d0 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c03a980 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.436+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c111740 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.437+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f4a6c039070 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.437+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c04b480 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.437+0000 7f4a78ff9640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f4a68034900 0x7f4a68036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.437+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4a6c067080 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.437+0000 7f4a73fff640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f4a68034900 0x7f4a68036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.438+0000 7f4a73fff640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f4a68034900 0x7f4a68036db0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006fd0 tx=0x7f4a64006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.442+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4a6c098df0 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.693+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4a7c1120a0 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.698+0000 7f4a78ff9640 1 -- 172.21.15.132:0/3013306685 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4a6c037030 con 0x7f4a7c103360 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f4a68034900 msgr2=0x7f4a68036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:26.279 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f4a68034900 0x7f4a68036db0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006fd0 tx=0x7f4a64006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 msgr2=0x7f4a7c110b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c110b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009810 tx=0x7f4a6c02c040 comp rx=0 tx=0).stop 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 shutdown_connections 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f4a68034900 0x7f4a68036db0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 --2- 172.21.15.132:0/3013306685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a7c103360 0x7f4a7c110b90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 >> 172.21.15.132:0/3013306685 conn(0x7f4a7c0fbc20 msgr2=0x7f4a7c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 shutdown_connections 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr 2024-06-13T01:19:25.700+0000 7f4a81bb1640 1 -- 172.21.15.132:0/3013306685 wait complete. 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout:Or, if you are only running a single cluster on this host: 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.280 INFO:teuthology.orchestra.run.smithi132.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: ceph telemetry on 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout:For more information see: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:26.281 INFO:teuthology.orchestra.run.smithi132.stdout:Bootstrap complete. 2024-06-13T01:19:26.314 INFO:tasks.cephadm:Fetching config... 2024-06-13T01:19:26.314 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:19:26.314 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-13T01:19:26.330 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-13T01:19:26.330 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:19:26.331 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-13T01:19:26.388 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-13T01:19:26.388 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:19:26.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/keyring of=/dev/stdout 2024-06-13T01:19:26.452 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-13T01:19:26.453 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:19:26.453 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-13T01:19:26.509 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-13T01:19:26.509 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7143o7mu/808ba+eIYaYptqQZdlOukJNvzUPJt4nxXnvOX8IBXhy18cG1mWW7DW4kzdjW/8/g5MDVzm3S+JdDgv3a4ODksH7hO92qTdKQww+VMaI1BK2TSatOOpoVz2aaoiuOEf5AQ+lDAHnPAWzPLkAy2/PSmf9x+8X3nRFjDOe9R524T7KPqufLhcbzehHeGnyhT01ee1TKQJ/ANkJGtOszyu7p45JpxUETFCYeNdgwMoOjsyMebKiY3g2Nubt5SEAo77AAUT6nWlEr2E+Gul0uM6QALzoit8fpcmlm0GdAFfGn3UP/p/I6ZRAJWUsQH01t/zoPITl0moyOjXTWa9ILhi0NrKxp9UhBey/QLEerf7UL2CBKjIg5QIif4kSB3L2NWWpvXmJl93ZswXOnEB/7UrErptd1HiIpRpozLjYWEX7qe/0Y0bYgSgVUuAPs+x7xQQ+rLygnrfNCDEVyP0x4ZT6X5b0i/K4JOQHgf5LfuqWavrLYE4xshXLcCEE= ceph-ad8d4854-2922-11ef-bc9c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-13T01:19:26.600 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7143o7mu/808ba+eIYaYptqQZdlOukJNvzUPJt4nxXnvOX8IBXhy18cG1mWW7DW4kzdjW/8/g5MDVzm3S+JdDgv3a4ODksH7hO92qTdKQww+VMaI1BK2TSatOOpoVz2aaoiuOEf5AQ+lDAHnPAWzPLkAy2/PSmf9x+8X3nRFjDOe9R524T7KPqufLhcbzehHeGnyhT01ee1TKQJ/ANkJGtOszyu7p45JpxUETFCYeNdgwMoOjsyMebKiY3g2Nubt5SEAo77AAUT6nWlEr2E+Gul0uM6QALzoit8fpcmlm0GdAFfGn3UP/p/I6ZRAJWUsQH01t/zoPITl0moyOjXTWa9ILhi0NrKxp9UhBey/QLEerf7UL2CBKjIg5QIif4kSB3L2NWWpvXmJl93ZswXOnEB/7UrErptd1HiIpRpozLjYWEX7qe/0Y0bYgSgVUuAPs+x7xQQ+rLygnrfNCDEVyP0x4ZT6X5b0i/K4JOQHgf5LfuqWavrLYE4xshXLcCEE= ceph-ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:19:26.612 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7143o7mu/808ba+eIYaYptqQZdlOukJNvzUPJt4nxXnvOX8IBXhy18cG1mWW7DW4kzdjW/8/g5MDVzm3S+JdDgv3a4ODksH7hO92qTdKQww+VMaI1BK2TSatOOpoVz2aaoiuOEf5AQ+lDAHnPAWzPLkAy2/PSmf9x+8X3nRFjDOe9R524T7KPqufLhcbzehHeGnyhT01ee1TKQJ/ANkJGtOszyu7p45JpxUETFCYeNdgwMoOjsyMebKiY3g2Nubt5SEAo77AAUT6nWlEr2E+Gul0uM6QALzoit8fpcmlm0GdAFfGn3UP/p/I6ZRAJWUsQH01t/zoPITl0moyOjXTWa9ILhi0NrKxp9UhBey/QLEerf7UL2CBKjIg5QIif4kSB3L2NWWpvXmJl93ZswXOnEB/7UrErptd1HiIpRpozLjYWEX7qe/0Y0bYgSgVUuAPs+x7xQQ+rLygnrfNCDEVyP0x4ZT6X5b0i/K4JOQHgf5LfuqWavrLYE4xshXLcCEE= ceph-ad8d4854-2922-11ef-bc9c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-13T01:19:26.652 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7143o7mu/808ba+eIYaYptqQZdlOukJNvzUPJt4nxXnvOX8IBXhy18cG1mWW7DW4kzdjW/8/g5MDVzm3S+JdDgv3a4ODksH7hO92qTdKQww+VMaI1BK2TSatOOpoVz2aaoiuOEf5AQ+lDAHnPAWzPLkAy2/PSmf9x+8X3nRFjDOe9R524T7KPqufLhcbzehHeGnyhT01ee1TKQJ/ANkJGtOszyu7p45JpxUETFCYeNdgwMoOjsyMebKiY3g2Nubt5SEAo77AAUT6nWlEr2E+Gul0uM6QALzoit8fpcmlm0GdAFfGn3UP/p/I6ZRAJWUsQH01t/zoPITl0moyOjXTWa9ILhi0NrKxp9UhBey/QLEerf7UL2CBKjIg5QIif4kSB3L2NWWpvXmJl93ZswXOnEB/7UrErptd1HiIpRpozLjYWEX7qe/0Y0bYgSgVUuAPs+x7xQQ+rLygnrfNCDEVyP0x4ZT6X5b0i/K4JOQHgf5LfuqWavrLYE4xshXLcCEE= ceph-ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:19:26.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-13T01:19:26.916 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:19:26.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:26 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3013306685' entity='client.admin' 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- 172.21.15.132:0/880830797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 msgr2=0x7f26540feb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 --2- 172.21.15.132:0/880830797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540feb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2644004850 tx=0x7f2644009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- 172.21.15.132:0/880830797 shutdown_connections 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 --2- 172.21.15.132:0/880830797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540feb30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- 172.21.15.132:0/880830797 >> 172.21.15.132:0/880830797 conn(0x7f26540f65e0 msgr2=0x7f26540f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- 172.21.15.132:0/880830797 shutdown_connections 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- 172.21.15.132:0/880830797 wait complete. 2024-06-13T01:19:29.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 Processor -- start 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- start start 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540fc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.181+0000 7f2658b62640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26540fc9a0 con 0x7f2654065f20 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f265259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540fc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f265259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540fc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34846/0 (socket says 172.21.15.132:34846) 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f265259c640 1 -- 172.21.15.132:0/1835742825 learned_addr learned my addr 172.21.15.132:0/1835742825 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f265259c640 1 -- 172.21.15.132:0/1835742825 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2644005040 con 0x7f2654065f20 2024-06-13T01:19:29.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f265259c640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540fc460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2644004820 tx=0x7f2644009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:29.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2644032650 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2644032c80 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2644033d60 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26540faa30 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.182+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26540faf00 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.183+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265405ead0 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.183+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f2644030020 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.184+0000 7f264affd640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2640034950 0x7f2640036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.184+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2644065f50 con 0x7f2654065f20 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.184+0000 7f264bfff640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2640034950 0x7f2640036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:29.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.184+0000 7f264bfff640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2640034950 0x7f2640036e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f263c006fd0 tx=0x7f263c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:29.186 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.188+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2644032df0 con 0x7f2654065f20 2024-06-13T01:19:29.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.359+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f265404b480 con 0x7f2654065f20 2024-06-13T01:19:29.358 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:29.358 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:29.359 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:19:29.359 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:29.359 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi132", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-13T01:19:29.359 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi132", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-13T01:19:29.359 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:19:29.359 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:29 smithi132 ceph-mon[29330]: Deploying daemon crash.smithi132 on smithi132 2024-06-13T01:19:29.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.363+0000 7f264affd640 1 -- 172.21.15.132:0/1835742825 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f26440302d0 con 0x7f2654065f20 2024-06-13T01:19:29.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2640034950 msgr2=0x7f2640036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2640034950 0x7f2640036e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f263c006fd0 tx=0x7f263c006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 msgr2=0x7f26540fc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540fc460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2644004820 tx=0x7f2644009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 shutdown_connections 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2640034950 0x7f2640036e00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 --2- 172.21.15.132:0/1835742825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2654065f20 0x7f26540fc460 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 >> 172.21.15.132:0/1835742825 conn(0x7f26540f65e0 msgr2=0x7f26541030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 shutdown_connections 2024-06-13T01:19:29.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:29.366+0000 7f2658b62640 1 -- 172.21.15.132:0/1835742825 wait complete. 2024-06-13T01:19:30.188 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-13T01:19:30.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-13T01:19:30.709 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:30 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1835742825' entity='client.admin' 2024-06-13T01:19:31.807 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:19:33.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 -- 172.21.15.132:0/1349406863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 msgr2=0x7f61e00feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:33.412 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 --2- 172.21.15.132:0/1349406863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00feb90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f61d0004850 tx=0x7f61d0009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 -- 172.21.15.132:0/1349406863 shutdown_connections 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 --2- 172.21.15.132:0/1349406863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00feb90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 -- 172.21.15.132:0/1349406863 >> 172.21.15.132:0/1349406863 conn(0x7f61e00f65e0 msgr2=0x7f61e00f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 -- 172.21.15.132:0/1349406863 shutdown_connections 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.412+0000 7f61e569c640 1 -- 172.21.15.132:0/1349406863 wait complete. 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61e569c640 1 Processor -- start 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61e569c640 1 -- start start 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61e569c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00fc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61e569c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00fca00 con 0x7f61e0065ee0 2024-06-13T01:19:33.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61deffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00fc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61deffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00fc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52994/0 (socket says 172.21.15.132:52994) 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61deffd640 1 -- 172.21.15.132:0/2004140922 learned_addr learned my addr 172.21.15.132:0/2004140922 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61deffd640 1 -- 172.21.15.132:0/2004140922 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0005040 con 0x7f61e0065ee0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.413+0000 7f61deffd640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00fc4c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f61d0003080 tx=0x7f61d0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.414+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d00326c0 con 0x7f61e0065ee0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.414+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e00faa90 con 0x7f61e0065ee0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.414+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61d0032cf0 con 0x7f61e0065ee0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.414+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0033db0 con 0x7f61e0065ee0 2024-06-13T01:19:33.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.414+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e00faf60 con 0x7f61e0065ee0 2024-06-13T01:19:33.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.414+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f61d0030020 con 0x7f61e0065ee0 2024-06-13T01:19:33.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.415+0000 7f61c77fe640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f61c8034900 0x7f61c8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:33.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.415+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e01024a0 con 0x7f61e0065ee0 2024-06-13T01:19:33.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.415+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f61d00667f0 con 0x7f61e0065ee0 2024-06-13T01:19:33.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.415+0000 7f61de7fc640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f61c8034900 0x7f61c8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:33.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.415+0000 7f61de7fc640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f61c8034900 0x7f61c8036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f61d4006fd0 tx=0x7f61d4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:33.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.419+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f61d003c380 con 0x7f61e0065ee0 2024-06-13T01:19:33.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.596+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f61e01059e0 con 0x7f61c8034900 2024-06-13T01:19:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.599+0000 7f61c77fe640 1 -- 172.21.15.132:0/2004140922 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f61e01059e0 con 0x7f61c8034900 2024-06-13T01:19:33.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f61c8034900 msgr2=0x7f61c8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:33.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f61c8034900 0x7f61c8036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f61d4006fd0 tx=0x7f61d4006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 msgr2=0x7f61e00fc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00fc4c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f61d0003080 tx=0x7f61d0009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 shutdown_connections 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f61c8034900 0x7f61c8036db0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 --2- 172.21.15.132:0/2004140922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61e0065ee0 0x7f61e00fc4c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 >> 172.21.15.132:0/2004140922 conn(0x7f61e00f65e0 msgr2=0x7f61e01030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 shutdown_connections 2024-06-13T01:19:33.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:33.601+0000 7f61e569c640 1 -- 172.21.15.132:0/2004140922 wait complete. 2024-06-13T01:19:34.555 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-06-13T01:19:34.556 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:19:34.556 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-06-13T01:19:34.575 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:19:34.575 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T01:19:34.633 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-06-13T01:19:34.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch host add smithi155 2024-06-13T01:19:34.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:34.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:34.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:34.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:34.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:34.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:34.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:34 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:35.044 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:19:36.195 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:35 smithi132 ceph-mon[29330]: Deploying daemon node-exporter.smithi132 on smithi132 2024-06-13T01:19:36.196 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:35 smithi132 ceph-mon[29330]: from='client.14186 -' entity='client.crash.smithi132' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:36.196 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:35 smithi132 ceph-mon[29330]: from='client.14188 -' entity='client.crash.smithi132' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:36.196 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:35 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:36.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 -- 172.21.15.132:0/2941408795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 msgr2=0x7f48340fedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:36.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 --2- 172.21.15.132:0/2941408795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f48340fedf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4824004850 tx=0x7f4824009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:36.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 -- 172.21.15.132:0/2941408795 shutdown_connections 2024-06-13T01:19:36.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 --2- 172.21.15.132:0/2941408795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f48340fedf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:36.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 -- 172.21.15.132:0/2941408795 >> 172.21.15.132:0/2941408795 conn(0x7f48340f6820 msgr2=0x7f48340f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:36.980 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 -- 172.21.15.132:0/2941408795 shutdown_connections 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.912+0000 7f4839486640 1 -- 172.21.15.132:0/2941408795 wait complete. 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4839486640 1 Processor -- start 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4839486640 1 -- start start 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4839486640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f483410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4839486640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483410cd10 con 0x7f4834065310 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4832ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f483410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4832ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f483410c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53060/0 (socket says 172.21.15.132:53060) 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.913+0000 7f4832ffd640 1 -- 172.21.15.132:0/4169599138 learned_addr learned my addr 172.21.15.132:0/4169599138 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f4832ffd640 1 -- 172.21.15.132:0/4169599138 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824005040 con 0x7f4834065310 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f4832ffd640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f483410c7d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f48240098d0 tx=0x7f4824009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4824032650 con 0x7f4834065310 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4824032c80 con 0x7f4834065310 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483410cf10 con 0x7f4834065310 2024-06-13T01:19:36.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f482403dc50 con 0x7f4834065310 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.914+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483410d380 con 0x7f4834065310 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.915+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f4824030020 con 0x7f4834065310 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.915+0000 7f481b7fe640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f481c034900 0x7f481c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.915+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4824067c70 con 0x7f4834065310 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.915+0000 7f48327fc640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f481c034900 0x7f481c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.916+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48341026b0 con 0x7f4834065310 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.916+0000 7f48327fc640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f481c034900 0x7f481c036db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4828006fd0 tx=0x7f4828006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:36.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:36.920+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4824033420 con 0x7f4834065310 2024-06-13T01:19:37.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:37.119+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7f4834101ee0 con 0x7f481c034900 2024-06-13T01:19:38.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:38 smithi132 ceph-mon[29330]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:38.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:38 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:38.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:38 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:38.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:38 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:38.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:38 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:38.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:38 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:39.926 INFO:teuthology.orchestra.run.smithi132.stdout:Added host 'smithi155' with addr '172.21.15.155' 2024-06-13T01:19:39.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.926+0000 7f481b7fe640 1 -- 172.21.15.132:0/4169599138 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4834101ee0 con 0x7f481c034900 2024-06-13T01:19:39.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:39 smithi132 ceph-mon[29330]: Deploying cephadm binary to smithi155 2024-06-13T01:19:39.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:39 smithi132 ceph-mon[29330]: Deploying daemon alertmanager.smithi132 on smithi132 2024-06-13T01:19:39.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:39 smithi132 ceph-mon[29330]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:39.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.928+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f481c034900 msgr2=0x7f481c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:39.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f481c034900 0x7f481c036db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4828006fd0 tx=0x7f4828006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:39.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 msgr2=0x7f483410c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:39.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f483410c7d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f48240098d0 tx=0x7f4824009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:39.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 shutdown_connections 2024-06-13T01:19:39.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f481c034900 0x7f481c036db0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:39.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 --2- 172.21.15.132:0/4169599138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4834065310 0x7f483410c7d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:39.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 >> 172.21.15.132:0/4169599138 conn(0x7f48340f6820 msgr2=0x7f48340f7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:39.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 shutdown_connections 2024-06-13T01:19:39.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:39.929+0000 7f4839486640 1 -- 172.21.15.132:0/4169599138 wait complete. 2024-06-13T01:19:40.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:40 smithi132 ceph-mon[29330]: Health check failed: 2 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-13T01:19:40.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:40 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:40.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:40 smithi132 ceph-mon[29330]: Added host smithi155 2024-06-13T01:19:41.196 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch host ls --format=json 2024-06-13T01:19:41.458 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:19:41.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:41 smithi132 ceph-mon[29330]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:43.804 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:43 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:43.804 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:43 smithi132 ceph-mon[29330]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:44.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 -- 172.21.15.132:0/939879724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 msgr2=0x7fc1380fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:44.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 --2- 172.21.15.132:0/939879724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc1380fc320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc128004850 tx=0x7fc128009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:44.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 -- 172.21.15.132:0/939879724 shutdown_connections 2024-06-13T01:19:44.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 --2- 172.21.15.132:0/939879724 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc1380fc320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:44.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 -- 172.21.15.132:0/939879724 >> 172.21.15.132:0/939879724 conn(0x7fc1380f76f0 msgr2=0x7fc1380f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:44.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 -- 172.21.15.132:0/939879724 shutdown_connections 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.639+0000 7fc13d369640 1 -- 172.21.15.132:0/939879724 wait complete. 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc13d369640 1 Processor -- start 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc13d369640 1 -- start start 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc13d369640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc13810c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc13d369640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13810ca70 con 0x7fc1380fbf10 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc136ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc13810c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc136ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc13810c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:47488/0 (socket says 172.21.15.132:47488) 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.640+0000 7fc136ffd640 1 -- 172.21.15.132:0/1280868262 learned_addr learned my addr 172.21.15.132:0/1280868262 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:44.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc136ffd640 1 -- 172.21.15.132:0/1280868262 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128005040 con 0x7fc1380fbf10 2024-06-13T01:19:44.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc136ffd640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc13810c530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc128004820 tx=0x7fc128009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:44.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc128032650 con 0x7fc1380fbf10 2024-06-13T01:19:44.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13810cc70 con 0x7fc1380fbf10 2024-06-13T01:19:44.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc128032c80 con 0x7fc1380fbf10 2024-06-13T01:19:44.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc12803eda0 con 0x7fc1380fbf10 2024-06-13T01:19:44.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.641+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13810d800 con 0x7fc1380fbf10 2024-06-13T01:19:44.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.642+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fc128030020 con 0x7fc1380fbf10 2024-06-13T01:19:44.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.642+0000 7fc12ffff640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fc124034900 0x7fc124036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:44.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.642+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc128066970 con 0x7fc1380fbf10 2024-06-13T01:19:44.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.642+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc138102480 con 0x7fc1380fbf10 2024-06-13T01:19:44.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.642+0000 7fc12e7fc640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fc124034900 0x7fc124036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:44.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.643+0000 7fc12e7fc640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fc124034900 0x7fc124036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc120006fd0 tx=0x7fc120006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:44.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.647+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc128033420 con 0x7fc1380fbf10 2024-06-13T01:19:44.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.821+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc13810da40 con 0x7fc124034900 2024-06-13T01:19:44.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.822+0000 7fc12ffff640 1 -- 172.21.15.132:0/1280868262 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fc13810da40 con 0x7fc124034900 2024-06-13T01:19:44.821 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:19:44.821 INFO:teuthology.orchestra.run.smithi132.stdout:[{"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-06-13T01:19:44.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fc124034900 msgr2=0x7fc124036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fc124034900 0x7fc124036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc120006fd0 tx=0x7fc120006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 msgr2=0x7fc13810c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc13810c530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc128004820 tx=0x7fc128009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 shutdown_connections 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fc124034900 0x7fc124036db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 --2- 172.21.15.132:0/1280868262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1380fbf10 0x7fc13810c530 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 >> 172.21.15.132:0/1280868262 conn(0x7fc1380f76f0 msgr2=0x7fc1380f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 shutdown_connections 2024-06-13T01:19:44.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:44.824+0000 7fc13d369640 1 -- 172.21.15.132:0/1280868262 wait complete. 2024-06-13T01:19:45.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:45 smithi132 ceph-mon[29330]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:45.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:45 smithi132 ceph-mon[29330]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T01:19:46.458 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-13T01:19:46.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd crush tunables default 2024-06-13T01:19:46.977 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:19:48.099 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:47 smithi132 ceph-mon[29330]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 -- 172.21.15.132:0/75034552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 msgr2=0x7fe6980fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 --2- 172.21.15.132:0/75034552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980fc340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe688004850 tx=0x7fe688009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 -- 172.21.15.132:0/75034552 shutdown_connections 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 --2- 172.21.15.132:0/75034552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980fc340 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 -- 172.21.15.132:0/75034552 >> 172.21.15.132:0/75034552 conn(0x7fe6980f7710 msgr2=0x7fe6980f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 -- 172.21.15.132:0/75034552 shutdown_connections 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.558+0000 7fe69e2da640 1 -- 172.21.15.132:0/75034552 wait complete. 2024-06-13T01:19:48.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe69e2da640 1 Processor -- start 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe69e2da640 1 -- start start 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe69e2da640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe69e2da640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe698071730 con 0x7fe6980fbf30 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe697fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe697fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:47510/0 (socket says 172.21.15.132:47510) 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.559+0000 7fe697fff640 1 -- 172.21.15.132:0/2231819797 learned_addr learned my addr 172.21.15.132:0/2231819797 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe697fff640 1 -- 172.21.15.132:0/2231819797 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688005040 con 0x7fe6980fbf30 2024-06-13T01:19:48.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe697fff640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980711f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe688004820 tx=0x7fe688009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:48.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe688032650 con 0x7fe6980fbf30 2024-06-13T01:19:48.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe688032c80 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69806f7c0 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe68803eda0 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.560+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69806fc90 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.561+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fe688030020 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.561+0000 7fe694ff9640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe680034900 0x7fe680036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.561+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69805ead0 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.561+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe6880668d0 con 0x7fe6980fbf30 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.561+0000 7fe6977fe640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe680034900 0x7fe680036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:48.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.562+0000 7fe6977fe640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe680034900 0x7fe680036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe68c006fd0 tx=0x7fe68c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:48.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.566+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe6880327b0 con 0x7fe6980fbf30 2024-06-13T01:19:48.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.747+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe69804b480 con 0x7fe6980fbf30 2024-06-13T01:19:48.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.790+0000 7fe694ff9640 1 -- 172.21.15.132:0/2231819797 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe68803c330 con 0x7fe6980fbf30 2024-06-13T01:19:48.789 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.792+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe680034900 msgr2=0x7fe680036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.792+0000 7fe69e2da640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe680034900 0x7fe680036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe68c006fd0 tx=0x7fe68c006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.792+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 msgr2=0x7fe6980711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.792+0000 7fe69e2da640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980711f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe688004820 tx=0x7fe688009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.792+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 shutdown_connections 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.792+0000 7fe69e2da640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe680034900 0x7fe680036db0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:48.791 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.793+0000 7fe69e2da640 1 --2- 172.21.15.132:0/2231819797 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6980fbf30 0x7fe6980711f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:48.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.793+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 >> 172.21.15.132:0/2231819797 conn(0x7fe6980f7710 msgr2=0x7fe6981030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:48.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.793+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 shutdown_connections 2024-06-13T01:19:48.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:19:48.793+0000 7fe69e2da640 1 -- 172.21.15.132:0/2231819797 wait complete. 2024-06-13T01:19:49.081 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:48 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2231819797' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-13T01:19:50.074 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:50.074 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2231819797' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: osdmap e4: 0 total, 0 up, 0 in 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-13T01:19:50.075 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:49 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:50.098 INFO:tasks.cephadm:Adding mon.smithi132 on smithi132 2024-06-13T01:19:50.098 INFO:tasks.cephadm:Adding mon.smithi155 on smithi155 2024-06-13T01:19:50.099 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch apply mon '2;smithi132:172.21.15.132=smithi132;smithi155:172.21.15.155=smithi155' 2024-06-13T01:19:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:50.396 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:51.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:50 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-13T01:19:51.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:50 smithi132 ceph-mon[29330]: Deploying daemon grafana.smithi132 on smithi132 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 -- 172.21.15.155:0/4114961388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 msgr2=0x7f800c0febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 --2- 172.21.15.155:0/4114961388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c0febb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8008004850 tx=0x7f8008009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 -- 172.21.15.155:0/4114961388 shutdown_connections 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 --2- 172.21.15.155:0/4114961388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c0febb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 -- 172.21.15.155:0/4114961388 >> 172.21.15.155:0/4114961388 conn(0x7f800c0f65e0 msgr2=0x7f800c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 -- 172.21.15.155:0/4114961388 shutdown_connections 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 -- 172.21.15.155:0/4114961388 wait complete. 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 Processor -- start 2024-06-13T01:19:51.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.874+0000 7f8013bbc640 1 -- start start 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8013bbc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c10c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8013bbc640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c10caa0 con 0x7f800c065ea0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8011958640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c10c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8011958640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c10c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:40248/0 (socket says 172.21.15.155:40248) 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8011958640 1 -- 172.21.15.155:0/2909880209 learned_addr learned my addr 172.21.15.155:0/2909880209 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8011958640 1 -- 172.21.15.155:0/2909880209 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008005040 con 0x7f800c065ea0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8011958640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c10c560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8008004820 tx=0x7f8008009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8008032650 con 0x7f800c065ea0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8008032c80 con 0x7f800c065ea0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c10cca0 con 0x7f800c065ea0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8008033d60 con 0x7f800c065ea0 2024-06-13T01:19:51.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.875+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c10d830 con 0x7f800c065ea0 2024-06-13T01:19:51.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.876+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f8008030020 con 0x7f800c065ea0 2024-06-13T01:19:51.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.877+0000 7f80027fc640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7ff8034900 0x7f7ff8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:51.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.877+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c1024a0 con 0x7f800c065ea0 2024-06-13T01:19:51.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.877+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8008066700 con 0x7f800c065ea0 2024-06-13T01:19:51.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.877+0000 7f8011157640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7ff8034900 0x7f7ff8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:51.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.877+0000 7f8011157640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7ff8034900 0x7f7ff8036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc006fd0 tx=0x7f7ffc006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:51.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:51.884+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f800803c380 con 0x7f800c065ea0 2024-06-13T01:19:52.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.055+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 --> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi132:172.21.15.132=smithi132;smithi155:172.21.15.155=smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7f800c10db20 con 0x7f7ff8034900 2024-06-13T01:19:52.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.061+0000 7f80027fc640 1 -- 172.21.15.155:0/2909880209 <== mgr.14160 v2:172.21.15.132:6800/3829496564 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f800c10db20 con 0x7f7ff8034900 2024-06-13T01:19:52.060 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-06-13T01:19:52.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7ff8034900 msgr2=0x7f7ff8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:52.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7ff8034900 0x7f7ff8036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc006fd0 tx=0x7f7ffc006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:52.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 msgr2=0x7f800c10c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:52.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c10c560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8008004820 tx=0x7f8008009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:52.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 shutdown_connections 2024-06-13T01:19:52.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f7ff8034900 0x7f7ff8036db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:52.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 --2- 172.21.15.155:0/2909880209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f800c065ea0 0x7f800c10c560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:52.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 >> 172.21.15.155:0/2909880209 conn(0x7f800c0f65e0 msgr2=0x7f800c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:52.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 shutdown_connections 2024-06-13T01:19:52.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:52.063+0000 7f8013bbc640 1 -- 172.21.15.155:0/2909880209 wait complete. 2024-06-13T01:19:52.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:51 smithi132 ceph-mon[29330]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:52.600 DEBUG:teuthology.orchestra.run.smithi155:mon.smithi155> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi155.service 2024-06-13T01:19:52.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:19:52.603 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:19:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:52.925 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:53.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:53 smithi132 ceph-mon[29330]: from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi132:172.21.15.132=smithi132;smithi155:172.21.15.155=smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:19:53.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:53 smithi132 ceph-mon[29330]: Saving service mon spec with placement smithi132:172.21.15.132=smithi132;smithi155:172.21.15.155=smithi155;count:2 2024-06-13T01:19:53.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:53 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:53.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:53 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:19:53.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:53 smithi132 ceph-mon[29330]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- 172.21.15.155:0/3169227528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 msgr2=0x7fa2980fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 --2- 172.21.15.155:0/3169227528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa2980fb450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa288004850 tx=0x7fa288009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- 172.21.15.155:0/3169227528 shutdown_connections 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 --2- 172.21.15.155:0/3169227528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa2980fb450 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- 172.21.15.155:0/3169227528 >> 172.21.15.155:0/3169227528 conn(0x7fa2980f6820 msgr2=0x7fa2980f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- 172.21.15.155:0/3169227528 shutdown_connections 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- 172.21.15.155:0/3169227528 wait complete. 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 Processor -- start 2024-06-13T01:19:54.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- start start 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa298065230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.340+0000 7fa29dd10640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29810f480 con 0x7fa2980fb040 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa2977fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa298065230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa2977fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa298065230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:40256/0 (socket says 172.21.15.155:40256) 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa2977fe640 1 -- 172.21.15.155:0/246040293 learned_addr learned my addr 172.21.15.155:0/246040293 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa2977fe640 1 -- 172.21.15.155:0/246040293 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288005040 con 0x7fa2980fb040 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa2977fe640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa298065230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa288004820 tx=0x7fa288009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa288032650 con 0x7fa2980fb040 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29810f5c0 con 0x7fa2980fb040 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa288032c80 con 0x7fa2980fb040 2024-06-13T01:19:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa28803eda0 con 0x7fa2980fb040 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.341+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29810fa30 con 0x7fa2980fb040 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.342+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fa288030020 con 0x7fa2980fb040 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.342+0000 7fa27ffff640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fa280034900 0x7fa280036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.342+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29805ead0 con 0x7fa2980fb040 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.342+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa2880669e0 con 0x7fa2980fb040 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.342+0000 7fa296ffd640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fa280034900 0x7fa280036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.343+0000 7fa296ffd640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fa280034900 0x7fa280036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa28c006fd0 tx=0x7fa28c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.347+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa28803c510 con 0x7fa2980fb040 2024-06-13T01:19:54.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.572+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa29804b480 con 0x7fa2980fb040 2024-06-13T01:19:54.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.573+0000 7fa27ffff640 1 -- 172.21.15.155:0/246040293 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fa28804c300 con 0x7fa2980fb040 2024-06-13T01:19:54.571 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:19:54.571 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:19:54.572 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:19:54.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fa280034900 msgr2=0x7fa280036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:54.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fa280034900 0x7fa280036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa28c006fd0 tx=0x7fa28c006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:54.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 msgr2=0x7fa298065230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:54.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa298065230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa288004820 tx=0x7fa288009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:54.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 shutdown_connections 2024-06-13T01:19:54.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fa280034900 0x7fa280036db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:54.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 --2- 172.21.15.155:0/246040293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2980fb040 0x7fa298065230 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:54.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 >> 172.21.15.155:0/246040293 conn(0x7fa2980f6820 msgr2=0x7fa2981021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:54.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 shutdown_connections 2024-06-13T01:19:54.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:54.575+0000 7fa29dd10640 1 -- 172.21.15.155:0/246040293 wait complete. 2024-06-13T01:19:54.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:54 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/246040293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:19:55.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:55 smithi132 ceph-mon[29330]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:56.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:19:56.065 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:19:56.316 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:56.363 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:57.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.676+0000 7f3836449640 1 -- 172.21.15.155:0/4272357068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 msgr2=0x7f38300fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:57.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.676+0000 7f3836449640 1 --2- 172.21.15.155:0/4272357068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f38300fc340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3820004850 tx=0x7f3820009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:57.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.676+0000 7f3836449640 1 -- 172.21.15.155:0/4272357068 shutdown_connections 2024-06-13T01:19:57.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.676+0000 7f3836449640 1 --2- 172.21.15.155:0/4272357068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f38300fc340 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:57.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.676+0000 7f3836449640 1 -- 172.21.15.155:0/4272357068 >> 172.21.15.155:0/4272357068 conn(0x7f38300f7710 msgr2=0x7f38300f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:57.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.677+0000 7f3836449640 1 -- 172.21.15.155:0/4272357068 shutdown_connections 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.677+0000 7f3836449640 1 -- 172.21.15.155:0/4272357068 wait complete. 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.677+0000 7f3836449640 1 Processor -- start 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.677+0000 7f3836449640 1 -- start start 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.677+0000 7f3836449640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f383010e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.677+0000 7f3836449640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010ec70 con 0x7f38300fbf30 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f383010e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f383010e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:40268/0 (socket says 172.21.15.155:40268) 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382ffff640 1 -- 172.21.15.155:0/2350586854 learned_addr learned my addr 172.21.15.155:0/2350586854 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:19:57.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382ffff640 1 -- 172.21.15.155:0/2350586854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820005040 con 0x7f38300fbf30 2024-06-13T01:19:57.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382ffff640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f383010e730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3820004820 tx=0x7f3820009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3820032650 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3820032c80 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383010ee70 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f382003eda0 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.678+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383010fa00 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.679+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f3820030020 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.679+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38300715d0 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.679+0000 7f382cff9640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3818034900 0x7f3818036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.679+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3820066b60 con 0x7f38300fbf30 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.680+0000 7f382f7fe640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3818034900 0x7f3818036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:19:57.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.680+0000 7f382f7fe640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3818034900 0x7f3818036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3824006fd0 tx=0x7f3824006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:19:57.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.684+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3820033420 con 0x7f38300fbf30 2024-06-13T01:19:57.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.912+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3830110400 con 0x7f38300fbf30 2024-06-13T01:19:57.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.912+0000 7f382cff9640 1 -- 172.21.15.155:0/2350586854 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f382003c330 con 0x7f38300fbf30 2024-06-13T01:19:57.911 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:19:57.911 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:19:57.911 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3818034900 msgr2=0x7f3818036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3818034900 0x7f3818036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3824006fd0 tx=0x7f3824006db0 comp rx=0 tx=0).stop 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 msgr2=0x7f383010e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f383010e730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3820004820 tx=0x7f3820009da0 comp rx=0 tx=0).stop 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 shutdown_connections 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3818034900 0x7f3818036db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 --2- 172.21.15.155:0/2350586854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38300fbf30 0x7f383010e730 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 >> 172.21.15.155:0/2350586854 conn(0x7f38300f7710 msgr2=0x7f38301052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 shutdown_connections 2024-06-13T01:19:57.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:19:57.914+0000 7f3836449640 1 -- 172.21.15.155:0/2350586854 wait complete. 2024-06-13T01:19:58.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:57 smithi132 ceph-mon[29330]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:19:59.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:58 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/2350586854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:19:59.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:19:59.517 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:19:59.773 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:19:59.820 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:00.103 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:19:59 smithi132 ceph-mon[29330]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:01.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/1836100973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 msgr2=0x7f5dc00fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:01.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 --2- 172.21.15.155:0/1836100973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc00fc840 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5db4004850 tx=0x7f5db4009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:01.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/1836100973 shutdown_connections 2024-06-13T01:20:01.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 --2- 172.21.15.155:0/1836100973 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc00fc840 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:01.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/1836100973 >> 172.21.15.155:0/1836100973 conn(0x7f5dc00f5d70 msgr2=0x7f5dc00f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:01.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/1836100973 shutdown_connections 2024-06-13T01:20:01.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/1836100973 wait complete. 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 Processor -- start 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.077+0000 7f5dc6ff8640 1 -- start start 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.078+0000 7f5dc6ff8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc010b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.078+0000 7f5dc6ff8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc010b9a0 con 0x7f5dc00fc430 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.078+0000 7f5dc4d94640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc010b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.078+0000 7f5dc4d94640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc010b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:43162/0 (socket says 172.21.15.155:43162) 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.078+0000 7f5dc4d94640 1 -- 172.21.15.155:0/2092981503 learned_addr learned my addr 172.21.15.155:0/2092981503 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.078+0000 7f5dc4d94640 1 -- 172.21.15.155:0/2092981503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db4005040 con 0x7f5dc00fc430 2024-06-13T01:20:01.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.079+0000 7f5dc4d94640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc010b460 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5db4004820 tx=0x7f5db4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.079+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5db4032650 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.079+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc010bba0 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.079+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5db4032c80 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.079+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc010c730 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.079+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5db403ed70 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.080+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f5db4030020 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.080+0000 7f5dbd7fa640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5db0034900 0x7f5db0036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.080+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5db4066960 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.080+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc010c070 con 0x7f5dc00fc430 2024-06-13T01:20:01.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.080+0000 7f5dbffff640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5db0034900 0x7f5db0036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:01.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.081+0000 7f5dbffff640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5db0034900 0x7f5db0036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5dac006fd0 tx=0x7f5dac006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:01.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.085+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5dc010c070 con 0x7f5dc00fc430 2024-06-13T01:20:01.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.311+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5dc010c070 con 0x7f5dc00fc430 2024-06-13T01:20:01.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.312+0000 7f5dbd7fa640 1 -- 172.21.15.155:0/2092981503 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5dc010c070 con 0x7f5dc00fc430 2024-06-13T01:20:01.311 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:01.311 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:01.311 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:01.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5db0034900 msgr2=0x7f5db0036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:01.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5db0034900 0x7f5db0036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5dac006fd0 tx=0x7f5dac006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 msgr2=0x7f5dc010b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc010b460 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5db4004820 tx=0x7f5db4009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 shutdown_connections 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f5db0034900 0x7f5db0036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 --2- 172.21.15.155:0/2092981503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dc00fc430 0x7f5dc010b460 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 >> 172.21.15.155:0/2092981503 conn(0x7f5dc00f5d70 msgr2=0x7f5dc0101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 shutdown_connections 2024-06-13T01:20:01.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:01.314+0000 7f5dc6ff8640 1 -- 172.21.15.155:0/2092981503 wait complete. 2024-06-13T01:20:02.051 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:01 smithi132 ceph-mon[29330]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:02.052 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:01 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/2092981503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:02.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:02.906 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:03.159 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:03.207 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:03.824 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:03 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:04.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- 172.21.15.155:0/810422638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 msgr2=0x7fca7c0fa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 --2- 172.21.15.155:0/810422638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0fa9d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fca6c004850 tx=0x7fca6c009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- 172.21.15.155:0/810422638 shutdown_connections 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 --2- 172.21.15.155:0/810422638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0fa9d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- 172.21.15.155:0/810422638 >> 172.21.15.155:0/810422638 conn(0x7fca7c0f5da0 msgr2=0x7fca7c0f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- 172.21.15.155:0/810422638 shutdown_connections 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- 172.21.15.155:0/810422638 wait complete. 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 Processor -- start 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- start start 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.624+0000 7fca80f43640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c10f2d0 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca7a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:04.627 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca7a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:43170/0 (socket says 172.21.15.155:43170) 2024-06-13T01:20:04.627 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca7a59c640 1 -- 172.21.15.155:0/1598873424 learned_addr learned my addr 172.21.15.155:0/1598873424 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:04.627 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca7a59c640 1 -- 172.21.15.155:0/1598873424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c005040 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca7a59c640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0f6c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fca6c004820 tx=0x7fca6c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:04.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca6c032650 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca6c032c80 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c10f410 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca6c03eda0 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.625+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c10f820 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.626+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fca6c030020 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.626+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c05ead0 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.626+0000 7fca72ffd640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fca68034900 0x7fca68036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.627+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fca6c066970 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.627+0000 7fca73fff640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fca68034900 0x7fca68036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:04.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.627+0000 7fca73fff640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fca68034900 0x7fca68036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fca64006fd0 tx=0x7fca64006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:04.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.632+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fca6c03c510 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.859+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca7c04b480 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.859+0000 7fca72ffd640 1 -- 172.21.15.155:0/1598873424 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fca6c03c330 con 0x7fca7c0fa5c0 2024-06-13T01:20:04.858 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:04.858 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:04.859 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:04.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fca68034900 msgr2=0x7fca68036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:04.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fca68034900 0x7fca68036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fca64006fd0 tx=0x7fca64006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 msgr2=0x7fca7c0f6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0f6c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fca6c004820 tx=0x7fca6c009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 shutdown_connections 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fca68034900 0x7fca68036db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 --2- 172.21.15.155:0/1598873424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fca7c0fa5c0 0x7fca7c0f6c60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 >> 172.21.15.155:0/1598873424 conn(0x7fca7c0f5da0 msgr2=0x7fca7c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 shutdown_connections 2024-06-13T01:20:04.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:04.861+0000 7fca80f43640 1 -- 172.21.15.155:0/1598873424 wait complete. 2024-06-13T01:20:04.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:04 smithi132 ceph-mon[29330]: Deploying daemon prometheus.smithi132 on smithi132 2024-06-13T01:20:06.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:05 smithi132 ceph-mon[29330]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:06.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:05 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/1598873424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:06.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:06.392 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:06.640 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:06.688 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:08.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 -- 172.21.15.155:0/2913600501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 msgr2=0x7f23bc0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 --2- 172.21.15.155:0/2913600501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc0fc320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009bd0 tx=0x7f23ac004d10 comp rx=0 tx=0).stop 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 -- 172.21.15.155:0/2913600501 shutdown_connections 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 --2- 172.21.15.155:0/2913600501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc0fc320 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 -- 172.21.15.155:0/2913600501 >> 172.21.15.155:0/2913600501 conn(0x7f23bc0f76f0 msgr2=0x7f23bc0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 -- 172.21.15.155:0/2913600501 shutdown_connections 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.054+0000 7f23c0a0e640 1 -- 172.21.15.155:0/2913600501 wait complete. 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23c0a0e640 1 Processor -- start 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23c0a0e640 1 -- start start 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23c0a0e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc10c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23c0a0e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc10ca50 con 0x7f23bc0fbf10 2024-06-13T01:20:08.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc10c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:08.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc10c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:43190/0 (socket says 172.21.15.155:43190) 2024-06-13T01:20:08.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.055+0000 7f23ba59c640 1 -- 172.21.15.155:0/3970640626 learned_addr learned my addr 172.21.15.155:0/3970640626 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23ba59c640 1 -- 172.21.15.155:0/3970640626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0098b0 con 0x7f23bc0fbf10 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23ba59c640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc10c510 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009ba0 tx=0x7f23ac004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23ac0329b0 con 0x7f23bc0fbf10 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23ac03d420 con 0x7f23bc0fbf10 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc10cc50 con 0x7f23bc0fbf10 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23ac031590 con 0x7f23bc0fbf10 2024-06-13T01:20:08.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.056+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc10d0a0 con 0x7f23bc0fbf10 2024-06-13T01:20:08.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.057+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f23ac02f030 con 0x7f23bc0fbf10 2024-06-13T01:20:08.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.057+0000 7f23aaffd640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f23a40348b0 0x7f23a4036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:08.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.057+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f23ac067d00 con 0x7f23bc0fbf10 2024-06-13T01:20:08.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.057+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc05ead0 con 0x7f23bc0fbf10 2024-06-13T01:20:08.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.057+0000 7f23b9d9b640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f23a40348b0 0x7f23a4036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:08.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.058+0000 7f23b9d9b640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f23a40348b0 0x7f23a4036d60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f23b0006fd0 tx=0x7f23b0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:08.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.062+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f23ac041070 con 0x7f23bc0fbf10 2024-06-13T01:20:08.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:07 smithi132 ceph-mon[29330]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:08.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:07 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:08.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.285+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23bc04b480 con 0x7f23bc0fbf10 2024-06-13T01:20:08.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.286+0000 7f23aaffd640 1 -- 172.21.15.155:0/3970640626 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f23ac04e710 con 0x7f23bc0fbf10 2024-06-13T01:20:08.285 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:08.285 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:08.285 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:08.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f23a40348b0 msgr2=0x7f23a4036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:08.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f23a40348b0 0x7f23a4036d60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f23b0006fd0 tx=0x7f23b0006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 msgr2=0x7f23bc10c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc10c510 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009ba0 tx=0x7f23ac004d10 comp rx=0 tx=0).stop 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 shutdown_connections 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f23a40348b0 0x7f23a4036d60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 --2- 172.21.15.155:0/3970640626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23bc0fbf10 0x7f23bc10c510 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 >> 172.21.15.155:0/3970640626 conn(0x7f23bc0f76f0 msgr2=0x7f23bc1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 shutdown_connections 2024-06-13T01:20:08.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:08.288+0000 7f23c0a0e640 1 -- 172.21.15.155:0/3970640626 wait complete. 2024-06-13T01:20:09.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:08 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/3970640626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:09.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:09.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:10.098 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:10.145 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:10.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:09 smithi132 ceph-mon[29330]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:11.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.416+0000 7fe4d6973640 1 -- 172.21.15.155:0/1694160160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 msgr2=0x7fe4d00fda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:11.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.416+0000 7fe4d6973640 1 --2- 172.21.15.155:0/1694160160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d00fda50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0004850 tx=0x7fe4c0009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:11.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 -- 172.21.15.155:0/1694160160 shutdown_connections 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 --2- 172.21.15.155:0/1694160160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d00fda50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 -- 172.21.15.155:0/1694160160 >> 172.21.15.155:0/1694160160 conn(0x7fe4d00f5da0 msgr2=0x7fe4d00f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 -- 172.21.15.155:0/1694160160 shutdown_connections 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 -- 172.21.15.155:0/1694160160 wait complete. 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 Processor -- start 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.417+0000 7fe4d6973640 1 -- start start 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4d6973640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d010c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4d6973640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d010ca70 con 0x7fe4d00fd640 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d010c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d010c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:37648/0 (socket says 172.21.15.155:37648) 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4cffff640 1 -- 172.21.15.155:0/4275072625 learned_addr learned my addr 172.21.15.155:0/4275072625 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4cffff640 1 -- 172.21.15.155:0/4275072625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c0005040 con 0x7fe4d00fd640 2024-06-13T01:20:11.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4cffff640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d010c530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098d0 tx=0x7fe4c0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:11.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c0032650 con 0x7fe4d00fd640 2024-06-13T01:20:11.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4c0032c80 con 0x7fe4d00fd640 2024-06-13T01:20:11.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d010cc70 con 0x7fe4d00fd640 2024-06-13T01:20:11.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.418+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c0033c10 con 0x7fe4d00fd640 2024-06-13T01:20:11.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.419+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d010d0e0 con 0x7fe4d00fd640 2024-06-13T01:20:11.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.420+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fe4c0030020 con 0x7fe4d00fd640 2024-06-13T01:20:11.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.420+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d005ead0 con 0x7fe4d00fd640 2024-06-13T01:20:11.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.420+0000 7fe4ccff9640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe4b8034900 0x7fe4b8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:11.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.420+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe4c0066620 con 0x7fe4d00fd640 2024-06-13T01:20:11.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.420+0000 7fe4cf7fe640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe4b8034900 0x7fe4b8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:11.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.420+0000 7fe4cf7fe640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe4b8034900 0x7fe4b8036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4006fd0 tx=0x7fe4c4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:11.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.425+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe4c00302d0 con 0x7fe4d00fd640 2024-06-13T01:20:11.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.652+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4d004b480 con 0x7fe4d00fd640 2024-06-13T01:20:11.651 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.653+0000 7fe4ccff9640 1 -- 172.21.15.155:0/4275072625 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fe4c0044b40 con 0x7fe4d00fd640 2024-06-13T01:20:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:11.652 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:11.652 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:11.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe4b8034900 msgr2=0x7fe4b8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:11.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe4b8034900 0x7fe4b8036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4006fd0 tx=0x7fe4c4006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:11.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 msgr2=0x7fe4d010c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:11.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d010c530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00098d0 tx=0x7fe4c0009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:11.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 shutdown_connections 2024-06-13T01:20:11.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7fe4b8034900 0x7fe4b8036db0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:11.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 --2- 172.21.15.155:0/4275072625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4d00fd640 0x7fe4d010c530 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:11.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 >> 172.21.15.155:0/4275072625 conn(0x7fe4d00f5da0 msgr2=0x7fe4d01030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:11.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 shutdown_connections 2024-06-13T01:20:11.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:11.655+0000 7fe4d6973640 1 -- 172.21.15.155:0/4275072625 wait complete. 2024-06-13T01:20:12.154 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:11 smithi132 ceph-mon[29330]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:12.154 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:11 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/4275072625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:13.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:13.242 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:13.492 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:13.540 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:13.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:13 smithi132 ceph-mon[29330]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:14.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:14 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:14.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:14 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:14.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:14 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:14.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:14 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.030+0000 7f3ee8f59640 1 -- 172.21.15.155:0/4183739510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 msgr2=0x7f3ee40fb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.030+0000 7f3ee8f59640 1 --2- 172.21.15.155:0/4183739510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee40fb270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4004850 tx=0x7f3ed4009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.030+0000 7f3ee8f59640 1 -- 172.21.15.155:0/4183739510 shutdown_connections 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.030+0000 7f3ee8f59640 1 --2- 172.21.15.155:0/4183739510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee40fb270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.030+0000 7f3ee8f59640 1 -- 172.21.15.155:0/4183739510 >> 172.21.15.155:0/4183739510 conn(0x7f3ee40f65b0 msgr2=0x7f3ee40f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.031+0000 7f3ee8f59640 1 -- 172.21.15.155:0/4183739510 shutdown_connections 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.031+0000 7f3ee8f59640 1 -- 172.21.15.155:0/4183739510 wait complete. 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.031+0000 7f3ee8f59640 1 Processor -- start 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.031+0000 7f3ee8f59640 1 -- start start 2024-06-13T01:20:15.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.031+0000 7f3ee8f59640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee410b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.031+0000 7f3ee8f59640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee410b9a0 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee410b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee410b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:37676/0 (socket says 172.21.15.155:37676) 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee259c640 1 -- 172.21.15.155:0/786787707 learned_addr learned my addr 172.21.15.155:0/786787707 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee259c640 1 -- 172.21.15.155:0/786787707 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4005040 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee259c640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee410b460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4004820 tx=0x7f3ed4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ed4032650 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee410bba0 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ed4032c80 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ed4033d60 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.032+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee410c730 con 0x7f3ee40fae60 2024-06-13T01:20:15.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.033+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 14) v1 ==== 42017+0+0 (secure 0 0 0) 0x7f3ed4030020 con 0x7f3ee40fae60 2024-06-13T01:20:15.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.033+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee4101380 con 0x7f3ee40fae60 2024-06-13T01:20:15.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.034+0000 7f3edaffd640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 0x7f3ed0036e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:15.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.034+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3ed4066820 con 0x7f3ee40fae60 2024-06-13T01:20:15.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.034+0000 7f3edbfff640 1 -- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 msgr2=0x7f3ed0036e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3829496564 2024-06-13T01:20:15.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.034+0000 7f3edbfff640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 0x7f3ed0036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:20:15.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.038+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3ed403c560 con 0x7f3ee40fae60 2024-06-13T01:20:15.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.235+0000 7f3edbfff640 1 -- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 msgr2=0x7f3ed0036e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3829496564 2024-06-13T01:20:15.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.235+0000 7f3edbfff640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 0x7f3ed0036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-13T01:20:15.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.268+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ee410d130 con 0x7f3ee40fae60 2024-06-13T01:20:15.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.268+0000 7f3edaffd640 1 -- 172.21.15.155:0/786787707 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f3ed403c380 con 0x7f3ee40fae60 2024-06-13T01:20:15.267 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:15.267 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:15.267 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 msgr2=0x7f3ed0036e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 0x7f3ed0036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 msgr2=0x7f3ee410b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee410b460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4004820 tx=0x7f3ed4009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 shutdown_connections 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f3ed00349a0 0x7f3ed0036e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 --2- 172.21.15.155:0/786787707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ee40fae60 0x7f3ee410b460 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 >> 172.21.15.155:0/786787707 conn(0x7f3ee40f65b0 msgr2=0x7f3ee4101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 shutdown_connections 2024-06-13T01:20:15.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:15.270+0000 7f3ee8f59640 1 -- 172.21.15.155:0/786787707 wait complete. 2024-06-13T01:20:16.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:15 smithi132 ceph-mon[29330]: from='mgr.14160 172.21.15.132:0/3555108017' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-13T01:20:16.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:15 smithi132 ceph-mon[29330]: mgrmap e14: smithi132.vjweoa(active, since 57s) 2024-06-13T01:20:16.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:15 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/786787707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:16.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:16.854 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:17.094 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:17.141 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:18.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 -- 172.21.15.155:0/2233501936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 msgr2=0x7f271c0fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:18.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 --2- 172.21.15.155:0/2233501936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c0fb280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2714004850 tx=0x7f2714009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:18.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 -- 172.21.15.155:0/2233501936 shutdown_connections 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 --2- 172.21.15.155:0/2233501936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c0fb280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 -- 172.21.15.155:0/2233501936 >> 172.21.15.155:0/2233501936 conn(0x7f271c0f65e0 msgr2=0x7f271c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 -- 172.21.15.155:0/2233501936 shutdown_connections 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.399+0000 7f2720c9a640 1 -- 172.21.15.155:0/2233501936 wait complete. 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f2720c9a640 1 Processor -- start 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f2720c9a640 1 -- start start 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f2720c9a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c065dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f2720c9a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c10f290 con 0x7f271c0fae70 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f271a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c065dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f271a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c065dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:37704/0 (socket says 172.21.15.155:37704) 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f271a59c640 1 -- 172.21.15.155:0/946779481 learned_addr learned my addr 172.21.15.155:0/946779481 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:18.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.400+0000 7f271a59c640 1 -- 172.21.15.155:0/946779481 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2714005040 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f271a59c640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c065dc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2714004820 tx=0x7f2714009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2714032650 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2714032c80 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c10f3d0 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2714033d60 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c10f8a0 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.401+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 14) v1 ==== 42017+0+0 (secure 0 0 0) 0x7f2714030020 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.402+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c05ead0 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.402+0000 7f2712ffd640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 0x7f2708036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.402+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2714066820 con 0x7f271c0fae70 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.402+0000 7f2719d9b640 1 -- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 msgr2=0x7f2708036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3829496564 2024-06-13T01:20:18.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.402+0000 7f2719d9b640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 0x7f2708036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:20:18.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.406+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f271403c560 con 0x7f271c0fae70 2024-06-13T01:20:18.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.603+0000 7f2719d9b640 1 -- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 msgr2=0x7f2708036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3829496564 2024-06-13T01:20:18.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.603+0000 7f2719d9b640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 0x7f2708036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-13T01:20:18.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.634+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f271c04b480 con 0x7f271c0fae70 2024-06-13T01:20:18.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.634+0000 7f2712ffd640 1 -- 172.21.15.155:0/946779481 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f271403c380 con 0x7f271c0fae70 2024-06-13T01:20:18.633 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:18.633 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:18.633 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 msgr2=0x7f2708036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 0x7f2708036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 msgr2=0x7f271c065dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c065dc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2714004820 tx=0x7f2714009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 shutdown_connections 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:6800/3829496564,v1:172.21.15.132:6801/3829496564] conn(0x7f2708034950 0x7f2708036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:18.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 --2- 172.21.15.155:0/946779481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f271c0fae70 0x7f271c065dc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:18.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 >> 172.21.15.155:0/946779481 conn(0x7f271c0f65e0 msgr2=0x7f271c101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:18.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 shutdown_connections 2024-06-13T01:20:18.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:18.636+0000 7f2720c9a640 1 -- 172.21.15.155:0/946779481 wait complete. 2024-06-13T01:20:18.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:18 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/946779481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:19.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: Active manager daemon smithi132.vjweoa restarted 2024-06-13T01:20:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: Activating manager daemon smithi132.vjweoa 2024-06-13T01:20:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: osdmap e5: 0 total, 0 up, 0 in 2024-06-13T01:20:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: mgrmap e15: smithi132.vjweoa(active, starting, since 0.00315802s) 2024-06-13T01:20:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:20:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr metadata", "who": "smithi132.vjweoa", "id": "smithi132.vjweoa"}]: dispatch 2024-06-13T01:20:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-13T01:20:19.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-13T01:20:19.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-13T01:20:19.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: Manager daemon smithi132.vjweoa is now available 2024-06-13T01:20:19.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:20:19.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:20:19.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:19.963 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:19.963 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:20:19.963 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/mirror_snapshot_schedule"}]: dispatch 2024-06-13T01:20:19.963 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.vjweoa/trash_purge_schedule"}]: dispatch 2024-06-13T01:20:20.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:20.211 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:20.465 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:20.517 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:21.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:21.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: [13/Jun/2024:01:20:20] ENGINE Bus STARTING 2024-06-13T01:20:21.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:21.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: mgrmap e16: smithi132.vjweoa(active, since 1.00519s) 2024-06-13T01:20:21.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: [13/Jun/2024:01:20:20] ENGINE Serving on https://172.21.15.132:7150 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: [13/Jun/2024:01:20:20] ENGINE Bus STARTED 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: [13/Jun/2024:01:20:20] ENGINE Error in HTTPServer.serve 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: Traceback (most recent call last): 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._connections.run(self.expiration_interval) 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._run(expiration_interval) 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: s = self.context.wrap_socket( 2024-06-13T01:20:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: return self.sslsocket_class._create( 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self.do_handshake() 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: self._sslobj.do_handshake() 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-13T01:20:21.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:22.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.203+0000 7faaa1eca640 1 -- 172.21.15.155:0/2597410253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 msgr2=0x7faa9c0fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:22.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.203+0000 7faaa1eca640 1 --2- 172.21.15.155:0/2597410253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c0fb280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa8c004850 tx=0x7faa8c009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:22.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 -- 172.21.15.155:0/2597410253 shutdown_connections 2024-06-13T01:20:22.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 --2- 172.21.15.155:0/2597410253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c0fb280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 -- 172.21.15.155:0/2597410253 >> 172.21.15.155:0/2597410253 conn(0x7faa9c0f65e0 msgr2=0x7faa9c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 -- 172.21.15.155:0/2597410253 shutdown_connections 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 -- 172.21.15.155:0/2597410253 wait complete. 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 Processor -- start 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 -- start start 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c10b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.204+0000 7faaa1eca640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c10b9b0 con 0x7faa9c0fae70 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa9b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c10b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa9b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c10b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:35576/0 (socket says 172.21.15.155:35576) 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa9b7fe640 1 -- 172.21.15.155:0/1100098458 learned_addr learned my addr 172.21.15.155:0/1100098458 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa9b7fe640 1 -- 172.21.15.155:0/1100098458 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c005040 con 0x7faa9c0fae70 2024-06-13T01:20:22.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa9b7fe640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c10b470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0098d0 tx=0x7faa8c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:22.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c008ed0 con 0x7faa9c0fae70 2024-06-13T01:20:22.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa8c0090d0 con 0x7faa9c0fae70 2024-06-13T01:20:22.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c10bbb0 con 0x7faa9c0fae70 2024-06-13T01:20:22.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c0338a0 con 0x7faa9c0fae70 2024-06-13T01:20:22.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.205+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c10bfc0 con 0x7faa9c0fae70 2024-06-13T01:20:22.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.206+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 41867+0+0 (secure 0 0 0) 0x7faa8c030020 con 0x7faa9c0fae70 2024-06-13T01:20:22.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.206+0000 7faa83fff640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faa84034790 0x7faa84036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:22.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.206+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7faa8c0660b0 con 0x7faa9c0fae70 2024-06-13T01:20:22.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.207+0000 7faa9affd640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faa84034790 0x7faa84036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:22.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.207+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c05ead0 con 0x7faa9c0fae70 2024-06-13T01:20:22.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.207+0000 7faa9affd640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faa84034790 0x7faa84036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faa90006fd0 tx=0x7faa90006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:22.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.211+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faa8c03c410 con 0x7faa9c0fae70 2024-06-13T01:20:22.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.225+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7faa8c039d50 con 0x7faa9c0fae70 2024-06-13T01:20:22.443 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.444+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa9c100bc0 con 0x7faa9c0fae70 2024-06-13T01:20:22.443 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.445+0000 7faa83fff640 1 -- 172.21.15.155:0/1100098458 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7faa8c03c410 con 0x7faa9c0fae70 2024-06-13T01:20:22.444 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:22.444 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:22.444 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faa84034790 msgr2=0x7faa84036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faa84034790 0x7faa84036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faa90006fd0 tx=0x7faa90006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 msgr2=0x7faa9c10b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c10b470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0098d0 tx=0x7faa8c009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 shutdown_connections 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faa84034790 0x7faa84036c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 --2- 172.21.15.155:0/1100098458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa9c0fae70 0x7faa9c10b470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 >> 172.21.15.155:0/1100098458 conn(0x7faa9c0f65e0 msgr2=0x7faa9c101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 shutdown_connections 2024-06-13T01:20:22.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:22.447+0000 7faaa1eca640 1 -- 172.21.15.155:0/1100098458 wait complete. 2024-06-13T01:20:22.625 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:22 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:23.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:23 smithi132 ceph-mon[29330]: mgrmap e17: smithi132.vjweoa(active, since 2s) 2024-06-13T01:20:23.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:23 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/1100098458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:24.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:24.110 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:24.373 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:24.434 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-13T01:20:24.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:24.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:24.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:24.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:24.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:20:26.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3340742132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 msgr2=0x7efe38101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:26.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 --2- 172.21.15.155:0/3340742132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38101210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efe28004850 tx=0x7efe28009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:26.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3340742132 shutdown_connections 2024-06-13T01:20:26.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 --2- 172.21.15.155:0/3340742132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38101210 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:26.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3340742132 >> 172.21.15.155:0/3340742132 conn(0x7efe38070e10 msgr2=0x7efe38071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:26.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3340742132 shutdown_connections 2024-06-13T01:20:26.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.569+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3340742132 wait complete. 2024-06-13T01:20:26.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe3e1d0640 1 Processor -- start 2024-06-13T01:20:26.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe3e1d0640 1 -- start start 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe3e1d0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38112620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe3e1d0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38112b60 con 0x7efe38100e40 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe377fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38112620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe377fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38112620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:35592/0 (socket says 172.21.15.155:35592) 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.570+0000 7efe377fe640 1 -- 172.21.15.155:0/3765754876 learned_addr learned my addr 172.21.15.155:0/3765754876 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe377fe640 1 -- 172.21.15.155:0/3765754876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe28005040 con 0x7efe38100e40 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe377fe640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38112620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efe28004820 tx=0x7efe28009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe2802cf10 con 0x7efe38100e40 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe28033b00 con 0x7efe38100e40 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe38112d60 con 0x7efe38100e40 2024-06-13T01:20:26.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe2803db70 con 0x7efe38100e40 2024-06-13T01:20:26.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.571+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe38113150 con 0x7efe38100e40 2024-06-13T01:20:26.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.572+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7efe28030020 con 0x7efe38100e40 2024-06-13T01:20:26.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.572+0000 7efe1ffff640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efe20034a40 0x7efe20036ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:26.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.572+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7efe280680d0 con 0x7efe38100e40 2024-06-13T01:20:26.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.572+0000 7efe36ffd640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efe20034a40 0x7efe20036ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:26.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.572+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe381122d0 con 0x7efe38100e40 2024-06-13T01:20:26.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.573+0000 7efe36ffd640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efe20034a40 0x7efe20036ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efe2c006fd0 tx=0x7efe2c006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:26.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.577+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efe2803de40 con 0x7efe38100e40 2024-06-13T01:20:26.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.834+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe381051c0 con 0x7efe38100e40 2024-06-13T01:20:26.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.835+0000 7efe1ffff640 1 -- 172.21.15.155:0/3765754876 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7efe28045390 con 0x7efe38100e40 2024-06-13T01:20:26.835 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:26.835 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:26.835 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:26.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.836+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efe20034a40 msgr2=0x7efe20036ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:26.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.836+0000 7efe3e1d0640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efe20034a40 0x7efe20036ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efe2c006fd0 tx=0x7efe2c006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:26.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.836+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 msgr2=0x7efe38112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:26.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.836+0000 7efe3e1d0640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38112620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efe28004820 tx=0x7efe28009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:26.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.837+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 shutdown_connections 2024-06-13T01:20:26.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.837+0000 7efe3e1d0640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efe20034a40 0x7efe20036ef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:26.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.837+0000 7efe3e1d0640 1 --2- 172.21.15.155:0/3765754876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38100e40 0x7efe38112620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:26.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.837+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 >> 172.21.15.155:0/3765754876 conn(0x7efe38070e10 msgr2=0x7efe38064730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:26.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.837+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 shutdown_connections 2024-06-13T01:20:26.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:26.837+0000 7efe3e1d0640 1 -- 172.21.15.155:0/3765754876 wait complete. 2024-06-13T01:20:26.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: Updating smithi132:/etc/ceph/ceph.conf 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: Updating smithi155:/etc/ceph/ceph.conf 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: Updating smithi155:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: Updating smithi132:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-13T01:20:26.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-13T01:20:26.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:27.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:27 smithi132 ceph-mon[29330]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-06-13T01:20:27.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:27 smithi132 ceph-mon[29330]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-06-13T01:20:27.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:27 smithi132 ceph-mon[29330]: Updating smithi155:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T01:20:27.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:27 smithi132 ceph-mon[29330]: Updating smithi132:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.client.admin.keyring 2024-06-13T01:20:27.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:27 smithi132 ceph-mon[29330]: Deploying daemon crash.smithi155 on smithi155 2024-06-13T01:20:27.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:27 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/3765754876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:28.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:28.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:28.820 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:30.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 -- 172.21.15.155:0/2644185385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 msgr2=0x7f75f81005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 --2- 172.21.15.155:0/2644185385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f81005c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f75f0004850 tx=0x7f75f0009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 -- 172.21.15.155:0/2644185385 shutdown_connections 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 --2- 172.21.15.155:0/2644185385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f81005c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 -- 172.21.15.155:0/2644185385 >> 172.21.15.155:0/2644185385 conn(0x7f75f8071520 msgr2=0x7f75f8071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 -- 172.21.15.155:0/2644185385 shutdown_connections 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 -- 172.21.15.155:0/2644185385 wait complete. 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.538+0000 7f75ff62c640 1 Processor -- start 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75ff62c640 1 -- start start 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75ff62c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f80fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75ff62c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80fe1e0 con 0x7f75f80649d0 2024-06-13T01:20:30.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75fd3c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f80fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75fd3c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f80fc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:40978/0 (socket says 172.21.15.155:40978) 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75fd3c8640 1 -- 172.21.15.155:0/2716808053 learned_addr learned my addr 172.21.15.155:0/2716808053 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.539+0000 7f75fd3c8640 1 -- 172.21.15.155:0/2716808053 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f0005040 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.540+0000 7f75fd3c8640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f80fc200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f75f0000c00 tx=0x7f75f0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.540+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f0033700 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.540+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f80fc7a0 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.540+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75f0033d30 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.540+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f80fcc70 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.540+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f0032e50 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.541+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f75f0030020 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.541+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f80fbd40 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.541+0000 7f75f5ffb640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f75e00349a0 0x7f75e0036e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.541+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f75f0066c70 con 0x7f75f80649d0 2024-06-13T01:20:30.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.541+0000 7f75fcbc7640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f75e00349a0 0x7f75e0036e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:30.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.542+0000 7f75fcbc7640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f75e00349a0 0x7f75e0036e50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00ad90 tx=0x7f75ec009410 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:30.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.546+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f75f0045700 con 0x7f75f80649d0 2024-06-13T01:20:30.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.779+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75f80fd8a0 con 0x7f75f80649d0 2024-06-13T01:20:30.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.780+0000 7f75f5ffb640 1 -- 172.21.15.155:0/2716808053 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f75f003a2d0 con 0x7f75f80649d0 2024-06-13T01:20:30.780 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:30.780 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:30.780 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f75e00349a0 msgr2=0x7f75e0036e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f75e00349a0 0x7f75e0036e50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00ad90 tx=0x7f75ec009410 comp rx=0 tx=0).stop 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 msgr2=0x7f75f80fc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f80fc200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f75f0000c00 tx=0x7f75f0009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 shutdown_connections 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f75e00349a0 0x7f75e0036e50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 --2- 172.21.15.155:0/2716808053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f75f80649d0 0x7f75f80fc200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 >> 172.21.15.155:0/2716808053 conn(0x7f75f8071520 msgr2=0x7f75f80fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:30.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 shutdown_connections 2024-06-13T01:20:30.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:30.782+0000 7f75ff62c640 1 -- 172.21.15.155:0/2716808053 wait complete. 2024-06-13T01:20:32.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:31 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/2716808053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:32.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:32.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:32.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:32.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:32.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:31 smithi132 ceph-mon[29330]: Deploying daemon node-exporter.smithi155 on smithi155 2024-06-13T01:20:32.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:32.373 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:32.625 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:35.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 -- 172.21.15.155:0/896362842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 msgr2=0x7f6050101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:35.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 --2- 172.21.15.155:0/896362842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f6050101210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f604c004850 tx=0x7f604c009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:35.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 -- 172.21.15.155:0/896362842 shutdown_connections 2024-06-13T01:20:35.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 --2- 172.21.15.155:0/896362842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f6050101210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:35.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 -- 172.21.15.155:0/896362842 >> 172.21.15.155:0/896362842 conn(0x7f60500f6a50 msgr2=0x7f60500f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 -- 172.21.15.155:0/896362842 shutdown_connections 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 -- 172.21.15.155:0/896362842 wait complete. 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 Processor -- start 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.400+0000 7f60585a5640 1 -- start start 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f60585a5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f605010d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f60585a5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605010da90 con 0x7f6050100e40 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6056341640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f605010d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6056341640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f605010d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:40996/0 (socket says 172.21.15.155:40996) 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6056341640 1 -- 172.21.15.155:0/3648544291 learned_addr learned my addr 172.21.15.155:0/3648544291 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6056341640 1 -- 172.21.15.155:0/3648544291 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c005040 con 0x7f6050100e40 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6056341640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f605010d550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f604c004820 tx=0x7f604c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:35.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f604c02cf10 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605010dc90 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f604c033b00 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.401+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f604c03db70 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.402+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605010e100 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.402+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f604c030020 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.402+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60500fc5b0 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.402+0000 7f6046ffd640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f603c0349f0 0x7f603c036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.402+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f604c0670c0 con 0x7f6050100e40 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.403+0000 7f6055b40640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f603c0349f0 0x7f603c036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.403+0000 7f6055b40640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f603c0349f0 0x7f603c036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6040006fd0 tx=0x7f6040006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:35.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.407+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f604c045570 con 0x7f6050100e40 2024-06-13T01:20:35.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.648+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f605004b480 con 0x7f6050100e40 2024-06-13T01:20:35.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.649+0000 7f6046ffd640 1 -- 172.21.15.155:0/3648544291 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f604c045390 con 0x7f6050100e40 2024-06-13T01:20:35.647 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:35.648 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:35.648 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f603c0349f0 msgr2=0x7f603c036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f603c0349f0 0x7f603c036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6040006fd0 tx=0x7f6040006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 msgr2=0x7f605010d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f605010d550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f604c004820 tx=0x7f604c009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 shutdown_connections 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f603c0349f0 0x7f603c036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 --2- 172.21.15.155:0/3648544291 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6050100e40 0x7f605010d550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 >> 172.21.15.155:0/3648544291 conn(0x7f60500f6a50 msgr2=0x7f60500f8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 shutdown_connections 2024-06-13T01:20:35.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:35.651+0000 7f60585a5640 1 -- 172.21.15.155:0/3648544291 wait complete. 2024-06-13T01:20:35.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:35 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:35.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:35 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:20:36.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:36.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:36.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:36.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.teuqcw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T01:20:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.teuqcw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-13T01:20:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T01:20:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: Deploying daemon mgr.smithi155.teuqcw on smithi155 2024-06-13T01:20:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:36 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/3648544291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:37.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:37.253 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:37.519 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:39.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 -- 172.21.15.155:0/3279784042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 msgr2=0x7f3880102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:39.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 --2- 172.21.15.155:0/3279784042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f3880102310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3874004850 tx=0x7f3874009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:39.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 -- 172.21.15.155:0/3279784042 shutdown_connections 2024-06-13T01:20:39.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 --2- 172.21.15.155:0/3279784042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f3880102310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:39.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 -- 172.21.15.155:0/3279784042 >> 172.21.15.155:0/3279784042 conn(0x7f3880070e10 msgr2=0x7f3880071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 -- 172.21.15.155:0/3279784042 shutdown_connections 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.449+0000 7f3886ec1640 1 -- 172.21.15.155:0/3279784042 wait complete. 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3886ec1640 1 Processor -- start 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3886ec1640 1 -- start start 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3886ec1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f38801125b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3886ec1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880112af0 con 0x7f3880101f40 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3884c5d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f38801125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3884c5d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f38801125b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.155:41028/0 (socket says 172.21.15.155:41028) 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.450+0000 7f3884c5d640 1 -- 172.21.15.155:0/3738115808 learned_addr learned my addr 172.21.15.155:0/3738115808 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f3884c5d640 1 -- 172.21.15.155:0/3738115808 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874005040 con 0x7f3880101f40 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f3884c5d640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f38801125b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3874004820 tx=0x7f3874009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:39.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387402cf10 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3874033b00 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3880112cf0 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387403db70 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.451+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3880113160 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.452+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f3874030020 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.452+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38800fd6b0 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.452+0000 7f387d7fa640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f38700349a0 0x7f3870036e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.453+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f387403d460 con 0x7f3880101f40 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.453+0000 7f387ffff640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f38700349a0 0x7f3870036e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.453+0000 7f387ffff640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f38700349a0 0x7f3870036e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f386c006fd0 tx=0x7f386c006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:39.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.458+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f387403b780 con 0x7f3880101f40 2024-06-13T01:20:39.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.721+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f388004b480 con 0x7f3880101f40 2024-06-13T01:20:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.722+0000 7f387d7fa640 1 -- 172.21.15.155:0/3738115808 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f387403c3c0 con 0x7f3880101f40 2024-06-13T01:20:39.720 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:39.720 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:18:23.169745Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-13T01:20:39.721 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-06-13T01:20:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f38700349a0 msgr2=0x7f3870036e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f38700349a0 0x7f3870036e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f386c006fd0 tx=0x7f386c006db0 comp rx=0 tx=0).stop 2024-06-13T01:20:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 msgr2=0x7f38801125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f38801125b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3874004820 tx=0x7f3874009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 shutdown_connections 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f38700349a0 0x7f3870036e50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 --2- 172.21.15.155:0/3738115808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3880101f40 0x7f38801125b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 >> 172.21.15.155:0/3738115808 conn(0x7f3880070e10 msgr2=0x7f3880103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 shutdown_connections 2024-06-13T01:20:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:39.724+0000 7f3886ec1640 1 -- 172.21.15.155:0/3738115808 wait complete. 2024-06-13T01:20:40.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:40.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:40.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:40.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:40.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T01:20:40.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:40.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: Deploying daemon mon.smithi155 on smithi155 2024-06-13T01:20:40.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:40.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:40.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:40 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/3738115808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:41.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-13T01:20:41.439 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mon dump -f json 2024-06-13T01:20:42.352 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:20:42.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:42 smithi132 ceph-mon[29330]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: Failed to load ceph-mgr modules: k8sevents 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: mon.smithi132 calling monitor election 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: mon.smithi155 calling monitor election 2024-06-13T01:20:49.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: mon.smithi132 is new leader, mons smithi132,smithi155 in quorum (ranks 0,1) 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: monmap e2: 2 mons at {smithi132=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],smithi155=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: fsmap 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: osdmap e5: 0 total, 0 up, 0 in 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: mgrmap e17: smithi132.vjweoa(active, since 29s) 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: Health detail: HEALTH_WARN 2 mgr modules have recently crashed 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: mgr module k8sevents crashed in daemon mgr.smithi132.vjweoa on host smithi132 at 2024-06-13T01:18:35.344960Z 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: mgr module k8sevents crashed in daemon mgr.smithi132.vjweoa on host smithi132 at 2024-06-13T01:18:44.853710Z 2024-06-13T01:20:49.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: Standby manager daemon smithi155.teuqcw started 2024-06-13T01:20:49.463 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:49.463 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:49.463 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:49.463 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:49.463 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:49.627 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:50.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: mgrmap e18: smithi132.vjweoa(active, since 29s), standbys: smithi155.teuqcw 2024-06-13T01:20:50.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr metadata", "who": "smithi155.teuqcw", "id": "smithi155.teuqcw"}]: dispatch 2024-06-13T01:20:50.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:20:50.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:50.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:50.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:20:50.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:50.484 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: mgrmap e18: smithi132.vjweoa(active, since 29s), standbys: smithi155.teuqcw 2024-06-13T01:20:50.484 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr metadata", "who": "smithi155.teuqcw", "id": "smithi155.teuqcw"}]: dispatch 2024-06-13T01:20:50.484 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:20:50.485 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:50.485 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:50.485 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:20:50.485 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:20:50.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/3751808101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 msgr2=0x7f9c040085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:50.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/3751808101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c040085c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9c180fc040 tx=0x7f9c14002040 comp rx=0 tx=0).stop 2024-06-13T01:20:50.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/3751808101 shutdown_connections 2024-06-13T01:20:50.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/3751808101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c040085c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:50.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/3751808101 >> 172.21.15.155:0/3751808101 conn(0x7f9c18070f20 msgr2=0x7f9c18071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:50.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/3751808101 shutdown_connections 2024-06-13T01:20:50.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.594+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/3751808101 wait complete. 2024-06-13T01:20:50.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.595+0000 7f9c1fcbd640 1 Processor -- start 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.595+0000 7f9c1fcbd640 1 -- start start 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.595+0000 7f9c1fcbd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c1810d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.595+0000 7f9c1fcbd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.595+0000 7f9c1fcbd640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c181169d0 con 0x7f9c04006160 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.595+0000 7f9c1fcbd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c18116b10 con 0x7f9c1810de10 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:35918/0 (socket says 172.21.15.155:35918) 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 -- 172.21.15.155:0/299139767 learned_addr learned my addr 172.21.15.155:0/299139767 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1da59640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c1810d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 -- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 msgr2=0x7f9c1810e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-13T01:20:50.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 -- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 msgr2=0x7f9c1810e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1d258640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1da59640 1 -- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 msgr2=0x7f9c1810e260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1da59640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1da59640 1 -- 172.21.15.155:0/299139767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14005040 con 0x7f9c04006160 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1da59640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c1810d8d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c14037040 tx=0x7f9c1402d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c14035a40 con 0x7f9c04006160 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c18116d10 con 0x7f9c04006160 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c1400f040 con 0x7f9c04006160 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.596+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c1403e7a0 con 0x7f9c04006160 2024-06-13T01:20:50.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.597+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c181171a0 con 0x7f9c04006160 2024-06-13T01:20:50.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.597+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c180fc6c0 con 0x7f9c04006160 2024-06-13T01:20:50.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.598+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f9c14040b60 con 0x7f9c04006160 2024-06-13T01:20:50.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.598+0000 7f9c0e7fc640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9bfc0654b0 0x7f9bfc067960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:50.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.598+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9c140a31f0 con 0x7f9c04006160 2024-06-13T01:20:50.599 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.599+0000 7f9c1d258640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9bfc0654b0 0x7f9bfc067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:50.599 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.599+0000 7f9c1d258640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9bfc0654b0 0x7f9bfc067960 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c08006fd0 tx=0x7f9c08008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:50.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.603+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c14072500 con 0x7f9c04006160 2024-06-13T01:20:50.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.835+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c1805ead0 con 0x7f9c04006160 2024-06-13T01:20:50.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.836+0000 7f9c0e7fc640 1 -- 172.21.15.155:0/299139767 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7f9c14031410 con 0x7f9c04006160 2024-06-13T01:20:50.835 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-06-13T01:20:50.835 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","modified":"2024-06-13T01:20:44.117151Z","created":"2024-06-13T01:18:23.169745Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi155","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-13T01:20:50.835 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9bfc0654b0 msgr2=0x7f9bfc067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9bfc0654b0 0x7f9bfc067960 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c08006fd0 tx=0x7f9c08008040 comp rx=0 tx=0).stop 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 msgr2=0x7f9c1810d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c1810d8d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c14037040 tx=0x7f9c1402d040 comp rx=0 tx=0).stop 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 shutdown_connections 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9bfc0654b0 0x7f9bfc067960 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c1810de10 0x7f9c1810e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 --2- 172.21.15.155:0/299139767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c04006160 0x7f9c1810d8d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 >> 172.21.15.155:0/299139767 conn(0x7f9c18070f20 msgr2=0x7f9c180fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 shutdown_connections 2024-06-13T01:20:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:20:50.838+0000 7f9c1fcbd640 1 -- 172.21.15.155:0/299139767 wait complete. 2024-06-13T01:20:51.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:51 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/299139767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:51.497 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:51 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/299139767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-13T01:20:51.522 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-13T01:20:51.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph config generate-minimal-conf 2024-06-13T01:20:51.803 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:20:52.839 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:52.839 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.839 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: Updating smithi132:/etc/ceph/ceph.conf 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: Updating smithi155:/etc/ceph/ceph.conf 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: Updating smithi155:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: Updating smithi132:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T01:20:52.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:52.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:52.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:52.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: Updating smithi132:/etc/ceph/ceph.conf 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: Updating smithi155:/etc/ceph/ceph.conf 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: Updating smithi155:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: Updating smithi132:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/config/ceph.conf 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T01:20:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 -- 172.21.15.132:0/2591818518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fcfb0 msgr2=0x7f2a600fd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 --2- 172.21.15.132:0/2591818518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fcfb0 0x7f2a600fd380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004850 tx=0x7f2a50009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 -- 172.21.15.132:0/2591818518 shutdown_connections 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 --2- 172.21.15.132:0/2591818518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fcfb0 0x7f2a600fd380 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 -- 172.21.15.132:0/2591818518 >> 172.21.15.132:0/2591818518 conn(0x7f2a600f6a70 msgr2=0x7f2a600f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 -- 172.21.15.132:0/2591818518 shutdown_connections 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.326+0000 7f2a6523a640 1 -- 172.21.15.132:0/2591818518 wait complete. 2024-06-13T01:20:53.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a6523a640 1 Processor -- start 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a6523a640 1 -- start start 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a6523a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a600fcfb0 0x7f2a600fae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a6523a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 0x7f2a600fb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a6523a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60118a40 con 0x7f2a600fb3b0 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a6523a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60118b80 con 0x7f2a600fcfb0 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 0x7f2a600fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 0x7f2a600fb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34574/0 (socket says 172.21.15.132:34574) 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.327+0000 7f2a5659c640 1 -- 172.21.15.132:0/65205543 learned_addr learned my addr 172.21.15.132:0/65205543 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a5ed9d640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a600fcfb0 0x7f2a600fae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a5659c640 1 -- 172.21.15.132:0/65205543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a600fcfb0 msgr2=0x7f2a600fae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a5659c640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a600fcfb0 0x7f2a600fae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a5659c640 1 -- 172.21.15.132:0/65205543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50005040 con 0x7f2a600fb3b0 2024-06-13T01:20:53.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a5659c640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 0x7f2a600fb800 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800b790 tx=0x7f2a480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a4800c910 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a4800be60 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a480135e0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a60118de0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.328+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a601192d0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.329+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a600657a0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.330+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f2a48014440 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.330+0000 7f2a57fff640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f2a4c065500 0x7f2a4c0679b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.330+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2a48080c70 con 0x7f2a600fb3b0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.330+0000 7f2a5ed9d640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f2a4c065500 0x7f2a4c0679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.330+0000 7f2a5ed9d640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f2a4c065500 0x7f2a4c0679b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a5002ab60 tx=0x7f2a50009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:20:53.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.334+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2a4804cdf0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.503+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2a6005ead0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.503+0000 7f2a57fff640 1 -- 172.21.15.132:0/65205543 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f2a480500b0 con 0x7f2a600fb3b0 2024-06-13T01:20:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:# minimal ceph.conf for ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:20:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-06-13T01:20:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = ad8d4854-2922-11ef-bc9c-c7b262605968 2024-06-13T01:20:53.503 INFO:teuthology.orchestra.run.smithi132.stdout: mon_host = [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-06-13T01:20:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.505+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f2a4c065500 msgr2=0x7f2a4c0679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.505+0000 7f2a6523a640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f2a4c065500 0x7f2a4c0679b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a5002ab60 tx=0x7f2a50009da0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.505+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 msgr2=0x7f2a600fb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.505+0000 7f2a6523a640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 0x7f2a600fb800 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800b790 tx=0x7f2a480091d0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 shutdown_connections 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f2a4c065500 0x7f2a4c0679b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a600fb3b0 0x7f2a600fb800 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 --2- 172.21.15.132:0/65205543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a600fcfb0 0x7f2a600fae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 >> 172.21.15.132:0/65205543 conn(0x7f2a600f6a70 msgr2=0x7f2a60101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 shutdown_connections 2024-06-13T01:20:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:20:53.506+0000 7f2a6523a640 1 -- 172.21.15.132:0/65205543 wait complete. 2024-06-13T01:20:53.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:53 smithi132 ceph-mon[29330]: Reconfiguring mon.smithi132 (unknown last config time)... 2024-06-13T01:20:53.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:53 smithi132 ceph-mon[29330]: Reconfiguring daemon mon.smithi132 on smithi132 2024-06-13T01:20:53.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:53 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/65205543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:53.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:53 smithi155 ceph-mon[35815]: Reconfiguring mon.smithi132 (unknown last config time)... 2024-06-13T01:20:53.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:53 smithi155 ceph-mon[35815]: Reconfiguring daemon mon.smithi132 on smithi132 2024-06-13T01:20:53.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:53 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/65205543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:54.795 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:54 smithi132 ceph-mon[29330]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:54.795 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:54.795 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:54.795 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi132.vjweoa", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T01:20:54.795 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T01:20:54.796 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:54 smithi155 ceph-mon[35815]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi132.vjweoa", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T01:20:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T01:20:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:55.005 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-13T01:20:55.006 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:20:55.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-06-13T01:20:55.034 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:20:55.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T01:20:55.099 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:20:55.099 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2024-06-13T01:20:55.132 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:20:55.132 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-13T01:20:55.201 INFO:tasks.cephadm:Deploying OSDs... 2024-06-13T01:20:55.201 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:20:55.201 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-06-13T01:20:55.216 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-13T01:20:55.217 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-06-13T01:20:55.272 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-13T01:20:55.272 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:55.272 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-13T01:20:55.272 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:55.272 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:55.273 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-06-13 01:20:13.144171159 +0000 2024-06-13T01:20:55.273 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-06-13 01:14:14.030063724 +0000 2024-06-13T01:20:55.273 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-06-13 01:14:14.030063724 +0000 2024-06-13T01:20:55.273 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-06-13 01:14:14.030063724 +0000 2024-06-13T01:20:55.273 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-13T01:20:55.336 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-06-13T01:20:55.336 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-06-13T01:20:55.337 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000207242 s, 2.5 MB/s 2024-06-13T01:20:55.338 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-13T01:20:55.396 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-06-13T01:20:55.451 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-13T01:20:55.451 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:55.451 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-06-13T01:20:55.452 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:55.452 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:55.452 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-06-13 01:20:13.144171159 +0000 2024-06-13T01:20:55.452 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-06-13 01:14:14.361055355 +0000 2024-06-13T01:20:55.452 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-06-13 01:14:14.361055355 +0000 2024-06-13T01:20:55.452 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-06-13 01:14:14.361055355 +0000 2024-06-13T01:20:55.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-13T01:20:55.515 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-06-13T01:20:55.515 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-06-13T01:20:55.516 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000192872 s, 2.7 MB/s 2024-06-13T01:20:55.517 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-13T01:20:55.574 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-06-13T01:20:55.629 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-13T01:20:55.629 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-06-13 01:20:13.144171159 +0000 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-06-13 01:14:14.700046783 +0000 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-06-13 01:14:14.700046783 +0000 2024-06-13T01:20:55.630 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-06-13 01:14:14.700046783 +0000 2024-06-13T01:20:55.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-13T01:20:55.690 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:55 smithi132 ceph-mon[29330]: Reconfiguring mgr.smithi132.vjweoa (unknown last config time)... 2024-06-13T01:20:55.690 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:55 smithi132 ceph-mon[29330]: Reconfiguring daemon mgr.smithi132.vjweoa on smithi132 2024-06-13T01:20:55.692 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-06-13T01:20:55.692 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-06-13T01:20:55.692 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000193508 s, 2.6 MB/s 2024-06-13T01:20:55.694 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-13T01:20:55.750 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-06-13T01:20:55.806 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-13T01:20:55.806 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-06-13 01:20:13.145171134 +0000 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-06-13 01:14:15.028038489 +0000 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-06-13 01:14:15.028038489 +0000 2024-06-13T01:20:55.807 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-06-13 01:14:15.028038489 +0000 2024-06-13T01:20:55.807 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-13T01:20:55.869 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-06-13T01:20:55.869 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-06-13T01:20:55.869 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000223674 s, 2.3 MB/s 2024-06-13T01:20:55.871 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-13T01:20:55.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:55 smithi155 ceph-mon[35815]: Reconfiguring mgr.smithi132.vjweoa (unknown last config time)... 2024-06-13T01:20:55.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:55 smithi155 ceph-mon[35815]: Reconfiguring daemon mgr.smithi132.vjweoa on smithi132 2024-06-13T01:20:55.927 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:20:55.928 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-06-13T01:20:55.947 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-13T01:20:55.947 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-06-13 01:20:42.736650521 +0000 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-06-13 01:14:14.039971167 +0000 2024-06-13T01:20:56.005 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-06-13 01:14:14.039971167 +0000 2024-06-13T01:20:56.006 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-06-13 01:14:14.039971167 +0000 2024-06-13T01:20:56.006 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-13T01:20:56.072 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-06-13T01:20:56.072 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-06-13T01:20:56.072 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000283393 s, 1.8 MB/s 2024-06-13T01:20:56.073 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-13T01:20:56.131 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-06-13 01:20:42.737650501 +0000 2024-06-13T01:20:56.188 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-06-13 01:14:14.379964183 +0000 2024-06-13T01:20:56.189 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-06-13 01:14:14.379964183 +0000 2024-06-13T01:20:56.189 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-06-13 01:14:14.379964183 +0000 2024-06-13T01:20:56.189 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-13T01:20:56.253 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-06-13T01:20:56.253 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-06-13T01:20:56.253 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000216724 s, 2.4 MB/s 2024-06-13T01:20:56.254 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-13T01:20:56.311 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-06-13 01:20:42.737650501 +0000 2024-06-13T01:20:56.367 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-06-13 01:14:14.713957322 +0000 2024-06-13T01:20:56.368 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-06-13 01:14:14.713957322 +0000 2024-06-13T01:20:56.368 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-06-13 01:14:14.713957322 +0000 2024-06-13T01:20:56.368 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-13T01:20:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-06-13T01:20:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-06-13T01:20:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000202924 s, 2.5 MB/s 2024-06-13T01:20:56.434 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-13T01:20:56.490 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-06-13T01:20:56.545 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-06-13 01:20:42.738650482 +0000 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-06-13 01:14:15.045950502 +0000 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-06-13 01:14:15.045950502 +0000 2024-06-13T01:20:56.546 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-06-13 01:14:15.045950502 +0000 2024-06-13T01:20:56.546 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-13T01:20:56.609 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-06-13T01:20:56.609 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-06-13T01:20:56.609 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000185796 s, 2.8 MB/s 2024-06-13T01:20:56.610 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-13T01:20:56.650 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:56 smithi155 ceph-mon[35815]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:56.650 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:56.650 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:56.650 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi132", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-13T01:20:56.650 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:56.666 INFO:tasks.cephadm:Deploying osd.0 on smithi132 with /dev/vg_nvme/lv_4... 2024-06-13T01:20:56.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-13T01:20:56.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:56 smithi132 ceph-mon[29330]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:56.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:56.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:56.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi132", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-13T01:20:56.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:20:57.533 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:20:57.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:57 smithi132 ceph-mon[29330]: Reconfiguring crash.smithi132 (monmap changed)... 2024-06-13T01:20:57.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:57 smithi132 ceph-mon[29330]: Reconfiguring daemon crash.smithi132 on smithi132 2024-06-13T01:20:57.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:57.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:57.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:57 smithi155 ceph-mon[35815]: Reconfiguring crash.smithi132 (monmap changed)... 2024-06-13T01:20:57.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:57 smithi155 ceph-mon[35815]: Reconfiguring daemon crash.smithi132 on smithi132 2024-06-13T01:20:57.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:57.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:20:58.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:58 smithi155 ceph-mon[35815]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:58.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:58 smithi155 ceph-mon[35815]: Reconfiguring alertmanager.smithi132 (dependencies changed)... 2024-06-13T01:20:58.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:20:58 smithi155 ceph-mon[35815]: Reconfiguring daemon alertmanager.smithi132 on smithi132 2024-06-13T01:20:58.890 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:58 smithi132 ceph-mon[29330]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:20:58.890 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:58 smithi132 ceph-mon[29330]: Reconfiguring alertmanager.smithi132 (dependencies changed)... 2024-06-13T01:20:58.891 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:20:58 smithi132 ceph-mon[29330]: Reconfiguring daemon alertmanager.smithi132 on smithi132 2024-06-13T01:20:59.547 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-13T01:20:59.628 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-13T01:20:59.641 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-06-13T01:20:59.641 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-06-13T01:20:59.641 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113669 s, 922 MB/s 2024-06-13T01:20:59.641 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-06-13T01:21:00.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:00 smithi155 ceph-mon[35815]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:00.903 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:00 smithi132 ceph-mon[29330]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:01.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-06-13T01:21:02.190 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:21:02.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:02 smithi155 ceph-mon[35815]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:02.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:02 smithi132 ceph-mon[29330]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- 172.21.15.132:0/1516695295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fb180 msgr2=0x7fa1100fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 --2- 172.21.15.132:0/1516695295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fb180 0x7fa1100fb590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1000048d0 tx=0x7fa100009da0 comp rx=0 tx=0).stop 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- 172.21.15.132:0/1516695295 shutdown_connections 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 --2- 172.21.15.132:0/1516695295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1100fc450 0x7fa1100fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 --2- 172.21.15.132:0/1516695295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fb180 0x7fa1100fb590 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- 172.21.15.132:0/1516695295 >> 172.21.15.132:0/1516695295 conn(0x7fa1100f6960 msgr2=0x7fa1100f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- 172.21.15.132:0/1516695295 shutdown_connections 2024-06-13T01:21:03.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- 172.21.15.132:0/1516695295 wait complete. 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 Processor -- start 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- start start 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1100fb180 0x7fa11010fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 0x7fa110112170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1100662e0 con 0x7fa1100fc450 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa115063640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa110066450 con 0x7fa1100fb180 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa107fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 0x7fa110112170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.777+0000 7fa107fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 0x7fa110112170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33082/0 (socket says 172.21.15.132:33082) 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa107fff640 1 -- 172.21.15.132:0/1634081977 learned_addr learned my addr 172.21.15.132:0/1634081977 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa107fff640 1 -- 172.21.15.132:0/1634081977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1100fb180 msgr2=0x7fa11010fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa107fff640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1100fb180 0x7fa11010fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa107fff640 1 -- 172.21.15.132:0/1634081977 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100005040 con 0x7fa1100fc450 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa107fff640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 0x7fa110112170 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b790 tx=0x7fa0f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:21:03.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f800c700 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0f800cd30 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa110112770 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f8013370 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.778+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa110112c90 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.779+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1100fa5e0 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.779+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fa0f8012070 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.780+0000 7fa1077fe640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa0fc0653b0 0x7fa0fc067860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.780+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa0f8081340 con 0x7fa1100fc450 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.780+0000 7fa10ed9d640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa0fc0653b0 0x7fa0fc067860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.780+0000 7fa10ed9d640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa0fc0653b0 0x7fa0fc067860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa10002a530 tx=0x7fa100009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:21:03.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.784+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa0f8050590 con 0x7fa1100fc450 2024-06-13T01:21:03.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:03.965+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa110100760 con 0x7fa0fc0653b0 2024-06-13T01:21:04.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:04 smithi132 ceph-mon[29330]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:04.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:21:04.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:21:04.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:04.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:04 smithi155 ceph-mon[35815]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:21:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:21:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:05.683 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:05 smithi132 ceph-mon[29330]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:21:05.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:05 smithi155 ceph-mon[35815]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:21:06.702 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:06 smithi132 ceph-mon[29330]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:06.702 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:06 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:06.702 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:06 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:06.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:06 smithi155 ceph-mon[35815]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:06.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:06 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:06.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:06 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:07.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:07 smithi132 ceph-mon[29330]: Reconfiguring grafana.smithi132 (dependencies changed)... 2024-06-13T01:21:07.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:07 smithi132 ceph-mon[29330]: Reconfiguring daemon grafana.smithi132 on smithi132 2024-06-13T01:21:07.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:07 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2000895135' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3479cf5-50de-49b8-8219-8373f5329897"}]: dispatch 2024-06-13T01:21:07.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:07 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2000895135' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3479cf5-50de-49b8-8219-8373f5329897"}]': finished 2024-06-13T01:21:07.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:07 smithi132 ceph-mon[29330]: osdmap e6: 1 total, 0 up, 1 in 2024-06-13T01:21:07.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:07 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:07 smithi155 ceph-mon[35815]: Reconfiguring grafana.smithi132 (dependencies changed)... 2024-06-13T01:21:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:07 smithi155 ceph-mon[35815]: Reconfiguring daemon grafana.smithi132 on smithi132 2024-06-13T01:21:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:07 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2000895135' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3479cf5-50de-49b8-8219-8373f5329897"}]: dispatch 2024-06-13T01:21:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:07 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2000895135' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3479cf5-50de-49b8-8219-8373f5329897"}]': finished 2024-06-13T01:21:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:07 smithi155 ceph-mon[35815]: osdmap e6: 1 total, 0 up, 1 in 2024-06-13T01:21:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:07 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:08.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:08 smithi132 ceph-mon[29330]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:08.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:08 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1008859383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:21:08.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:08 smithi155 ceph-mon[35815]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:08.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:08 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1008859383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:21:10.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:10 smithi155 ceph-mon[35815]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:10.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:10 smithi132 ceph-mon[29330]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:12.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:12 smithi155 ceph-mon[35815]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:12.922 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:12 smithi132 ceph-mon[29330]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:14.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:14 smithi155 ceph-mon[35815]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:14.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:14 smithi132 ceph-mon[29330]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:16.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:16 smithi132 ceph-mon[29330]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:17.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:16 smithi155 ceph-mon[35815]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:18.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:17 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:18.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:17 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:18.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:17 smithi132 ceph-mon[29330]: Reconfiguring prometheus.smithi132 (dependencies changed)... 2024-06-13T01:21:18.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:17 smithi132 ceph-mon[29330]: Reconfiguring daemon prometheus.smithi132 on smithi132 2024-06-13T01:21:18.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:17 smithi132 ceph-mon[29330]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:18.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:17 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:18.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:17 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:18.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:17 smithi155 ceph-mon[35815]: Reconfiguring prometheus.smithi132 (dependencies changed)... 2024-06-13T01:21:18.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:17 smithi155 ceph-mon[35815]: Reconfiguring daemon prometheus.smithi132 on smithi132 2024-06-13T01:21:18.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:17 smithi155 ceph-mon[35815]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:18.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:18 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-13T01:21:18.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:18 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:18.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:18 smithi132 ceph-mon[29330]: Deploying daemon osd.0 on smithi132 2024-06-13T01:21:19.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:18 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-13T01:21:19.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:18 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:19.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:18 smithi155 ceph-mon[35815]: Deploying daemon osd.0 on smithi132 2024-06-13T01:21:20.064 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:19 smithi132 ceph-mon[29330]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:20.064 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:20.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:19 smithi155 ceph-mon[35815]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:20.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:22.518 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:22 smithi132 ceph-mon[29330]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:22.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:22 smithi155 ceph-mon[35815]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:23.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:23.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:23.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:23 smithi132 ceph-mon[29330]: Reconfiguring crash.smithi155 (monmap changed)... 2024-06-13T01:21:23.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-13T01:21:23.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:23.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:23 smithi132 ceph-mon[29330]: Reconfiguring daemon crash.smithi155 on smithi155 2024-06-13T01:21:23.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:23.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:23.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:23 smithi155 ceph-mon[35815]: Reconfiguring crash.smithi155 (monmap changed)... 2024-06-13T01:21:23.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-13T01:21:23.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:23.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:23 smithi155 ceph-mon[35815]: Reconfiguring daemon crash.smithi155 on smithi155 2024-06-13T01:21:24.687 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:24 smithi155 ceph-mon[35815]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:24.687 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:24 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:24.687 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:24 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:24.687 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:24 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.teuqcw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T01:21:24.687 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:24 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T01:21:24.688 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:24 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:24.740 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:24 smithi132 ceph-mon[29330]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:24.740 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:24.740 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:24.740 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.teuqcw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-13T01:21:24.740 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-13T01:21:24.740 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:24 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: Reconfiguring mgr.smithi155.teuqcw (monmap changed)... 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: Reconfiguring daemon mgr.smithi155.teuqcw on smithi155 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T01:21:25.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:25 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:25.959 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: Reconfiguring mgr.smithi155.teuqcw (monmap changed)... 2024-06-13T01:21:25.960 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: Reconfiguring daemon mgr.smithi155.teuqcw on smithi155 2024-06-13T01:21:25.960 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:25.960 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:25.960 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-13T01:21:25.960 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-13T01:21:25.960 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:25 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:26.898 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:26 smithi132 ceph-mon[29330]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:26.899 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:26 smithi132 ceph-mon[29330]: Reconfiguring mon.smithi155 (monmap changed)... 2024-06-13T01:21:26.899 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:26 smithi132 ceph-mon[29330]: Reconfiguring daemon mon.smithi155 on smithi155 2024-06-13T01:21:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:26 smithi155 ceph-mon[35815]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:26 smithi155 ceph-mon[35815]: Reconfiguring mon.smithi155 (monmap changed)... 2024-06-13T01:21:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:26 smithi155 ceph-mon[35815]: Reconfiguring daemon mon.smithi155 on smithi155 2024-06-13T01:21:28.010 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.010 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.010 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-13T01:21:28.010 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-13T01:21:28.010 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi132.front.sepia.ceph.com:9093"}]: dispatch 2024-06-13T01:21:28.010 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi132.front.sepia.ceph.com:9093"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.011 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi132.front.sepia.ceph.com:9093"}]: dispatch 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi132.front.sepia.ceph.com:9093"}]: dispatch 2024-06-13T01:21:28.023 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi132.front.sepia.ceph.com:3000"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi132.front.sepia.ceph.com:9095"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.024 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:28 smithi132 ceph-mon[29330]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:28.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:28 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:28.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:28 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:29.041 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:28 smithi155 ceph-mon[35815]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:29.042 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:28 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:29.042 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:28 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:30.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:30 smithi132 ceph-mon[29330]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:31.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:30 smithi155 ceph-mon[35815]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:31.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.442+0000 7fa1077fe640 1 -- 172.21.15.132:0/1634081977 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa110100760 con 0x7fa0fc0653b0 2024-06-13T01:21:31.441 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 0 on host 'smithi132' 2024-06-13T01:21:31.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa0fc0653b0 msgr2=0x7fa0fc067860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:31.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa0fc0653b0 0x7fa0fc067860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa10002a530 tx=0x7fa100009da0 comp rx=0 tx=0).stop 2024-06-13T01:21:31.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 msgr2=0x7fa110112170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:31.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 0x7fa110112170 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b790 tx=0x7fa0f80091d0 comp rx=0 tx=0).stop 2024-06-13T01:21:31.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 shutdown_connections 2024-06-13T01:21:31.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa0fc0653b0 0x7fa0fc067860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:31.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1100fc450 0x7fa110112170 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:31.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 --2- 172.21.15.132:0/1634081977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1100fb180 0x7fa11010fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:31.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 >> 172.21.15.132:0/1634081977 conn(0x7fa1100f6960 msgr2=0x7fa1100f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:21:31.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 shutdown_connections 2024-06-13T01:21:31.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:31.444+0000 7fa115063640 1 -- 172.21.15.132:0/1634081977 wait complete. 2024-06-13T01:21:32.072 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:31 smithi132 ceph-mon[29330]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T01:21:32.072 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:32.072 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:32.096 DEBUG:teuthology.orchestra.run.smithi132:osd.0> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.0.service 2024-06-13T01:21:32.098 INFO:tasks.cephadm:Deploying osd.1 on smithi132 with /dev/vg_nvme/lv_3... 2024-06-13T01:21:32.098 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-13T01:21:32.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:31 smithi155 ceph-mon[35815]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-13T01:21:32.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:31 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:32.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:31 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:32.576 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:21:33.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:32 smithi155 ceph-mon[35815]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:33.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:32 smithi155 ceph-mon[35815]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T01:21:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:32 smithi155 ceph-mon[35815]: osdmap e7: 1 total, 0 up, 1 in 2024-06-13T01:21:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:32 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:32 smithi155 ceph-mon[35815]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-06-13T01:21:33.172 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-mon[29330]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:33.172 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-mon[29330]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-13T01:21:33.172 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-mon[29330]: osdmap e7: 1 total, 0 up, 1 in 2024-06-13T01:21:33.172 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:33.172 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-mon[29330]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-06-13T01:21:33.172 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0[47104]: 2024-06-13T01:21:32.769+0000 7fcb08228640 -1 osd.0 0 waiting for initial osdmap 2024-06-13T01:21:33.173 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:21:32 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0[47104]: 2024-06-13T01:21:32.772+0000 7fcb0105b640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T01:21:34.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:33 smithi155 ceph-mon[35815]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-06-13T01:21:34.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:33 smithi155 ceph-mon[35815]: osdmap e8: 1 total, 0 up, 1 in 2024-06-13T01:21:34.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:34.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:34.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:33 smithi132 ceph-mon[29330]: from='osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-06-13T01:21:34.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:33 smithi132 ceph-mon[29330]: osdmap e8: 1 total, 0 up, 1 in 2024-06-13T01:21:34.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:34.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: purged_snaps scrub starts 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: purged_snaps scrub ok 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] boot 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: osdmap e9: 1 total, 1 up, 1 in 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:34.904 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:35.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: purged_snaps scrub starts 2024-06-13T01:21:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: purged_snaps scrub ok 2024-06-13T01:21:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-13T01:21:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: osd.0 [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] boot 2024-06-13T01:21:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: osdmap e9: 1 total, 1 up, 1 in 2024-06-13T01:21:35.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-13T01:21:35.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:35.656 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-13T01:21:35.740 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-13T01:21:35.754 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-06-13T01:21:35.754 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-06-13T01:21:35.754 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130389 s, 804 MB/s 2024-06-13T01:21:35.754 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-06-13T01:21:35.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:35 smithi155 ceph-mon[35815]: osdmap e10: 1 total, 1 up, 1 in 2024-06-13T01:21:35.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:35 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:35.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:35 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:35.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:35 smithi132 ceph-mon[29330]: osdmap e10: 1 total, 1 up, 1 in 2024-06-13T01:21:35.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:35 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:35.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:35 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:36.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-06-13T01:21:37.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:36 smithi155 ceph-mon[35815]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:37.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:36 smithi132 ceph-mon[29330]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:37.743 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:21:38.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:38 smithi132 ceph-mon[29330]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:39.117 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:38 smithi155 ceph-mon[35815]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:39.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 -- 172.21.15.132:0/3567878380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 msgr2=0x7f7c8c0feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:39.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3567878380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c0feea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0048d0 tx=0x7f7c7c009da0 comp rx=0 tx=0).stop 2024-06-13T01:21:39.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 -- 172.21.15.132:0/3567878380 shutdown_connections 2024-06-13T01:21:39.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3567878380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c8c0ff3e0 0x7f7c8c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:39.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3567878380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c0feea0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:39.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 -- 172.21.15.132:0/3567878380 >> 172.21.15.132:0/3567878380 conn(0x7f7c8c0f6970 msgr2=0x7f7c8c0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 -- 172.21.15.132:0/3567878380 shutdown_connections 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 -- 172.21.15.132:0/3567878380 wait complete. 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 Processor -- start 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.937+0000 7f7c939fc640 1 -- start start 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c939fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c10cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c939fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c8c0ff3e0 0x7f7c8c10d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c939fc640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c10da70 con 0x7f7c8c0659a0 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c939fc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c10e290 con 0x7f7c8c0ff3e0 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c10cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c10cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53040/0 (socket says 172.21.15.132:53040) 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 -- 172.21.15.132:0/3400230994 learned_addr learned my addr 172.21.15.132:0/3400230994 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:21:39.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c90f97640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c8c0ff3e0 0x7f7c8c10d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 -- 172.21.15.132:0/3400230994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c8c0ff3e0 msgr2=0x7f7c8c10d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c8c0ff3e0 0x7f7c8c10d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 -- 172.21.15.132:0/3400230994 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c005040 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c91798640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c10cee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c000c00 tx=0x7f7c7c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7c038070 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8c10e4c0 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c7c032970 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.938+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7c0338e0 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.939+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c10ea00 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.940+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c0fa5f0 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.940+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f7c7c030020 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.940+0000 7f7c81ffb640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c780653a0 0x7f7c78067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.940+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f7c7c0a1770 con 0x7f7c8c0659a0 2024-06-13T01:21:39.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.940+0000 7f7c90f97640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c780653a0 0x7f7c78067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:21:39.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.941+0000 7f7c90f97640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c780653a0 0x7f7c78067850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c10cc90 tx=0x7f7c8800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:21:39.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:39.945+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7c7c070810 con 0x7f7c8c0659a0 2024-06-13T01:21:40.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:40.127+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c8c10ed50 con 0x7f7c780653a0 2024-06-13T01:21:40.331 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: Detected new or changed devices on smithi132 2024-06-13T01:21:40.331 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: Adjusting osd_memory_target on smithi132 to 12864M 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:21:40.332 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:40 smithi155 ceph-mon[35815]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:40.343 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: Detected new or changed devices on smithi132 2024-06-13T01:21:40.343 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: Adjusting osd_memory_target on smithi132 to 12864M 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:21:40.344 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:40 smithi132 ceph-mon[29330]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:41.159 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:41 smithi132 ceph-mon[29330]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:21:41.160 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:41 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:21:41.160 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:41 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:21:41.160 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:41 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:41.160 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:41 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:41.160 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:41 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:41.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:41 smithi155 ceph-mon[35815]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:21:41.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:41 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:21:41.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:41 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:21:41.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:41 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:41.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:41 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:41.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:41 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:42.367 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:42 smithi132 ceph-mon[29330]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:42.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:42 smithi155 ceph-mon[35815]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:44.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:44 smithi132 ceph-mon[29330]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:44 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3258113572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f55f5e-4836-40db-9b9c-712be934facf"}]: dispatch 2024-06-13T01:21:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:44 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3258113572' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f55f5e-4836-40db-9b9c-712be934facf"}]': finished 2024-06-13T01:21:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:44 smithi132 ceph-mon[29330]: osdmap e11: 2 total, 1 up, 2 in 2024-06-13T01:21:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:44 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:21:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:44 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1406317317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:21:44.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:44 smithi155 ceph-mon[35815]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:44.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:44 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3258113572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f55f5e-4836-40db-9b9c-712be934facf"}]: dispatch 2024-06-13T01:21:44.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:44 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3258113572' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f55f5e-4836-40db-9b9c-712be934facf"}]': finished 2024-06-13T01:21:44.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:44 smithi155 ceph-mon[35815]: osdmap e11: 2 total, 1 up, 2 in 2024-06-13T01:21:44.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:44 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:21:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:44 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1406317317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:21:46.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:46 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:46.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:46 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:46.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:46 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:46.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:46 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:21:46.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:46 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:46.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:46 smithi155 ceph-mon[35815]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:46.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:46 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:46.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:46 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:46.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:46 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:46.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:46 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:21:46.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:46 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:46.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:46 smithi132 ceph-mon[29330]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:48.590 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:48 smithi132 ceph-mon[29330]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:48.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:48 smithi155 ceph-mon[35815]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:49.677 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:49.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:21:50.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:50 smithi132 ceph-mon[29330]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:50.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:50 smithi155 ceph-mon[35815]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:51.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:51 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-13T01:21:51.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:51 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:51.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:51 smithi132 ceph-mon[29330]: Deploying daemon osd.1 on smithi132 2024-06-13T01:21:51.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:51 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-13T01:21:51.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:51 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:21:51.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:51 smithi155 ceph-mon[35815]: Deploying daemon osd.1 on smithi132 2024-06-13T01:21:52.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:52 smithi132 ceph-mon[29330]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:52.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:52 smithi155 ceph-mon[35815]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:54.653 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:54 smithi132 ceph-mon[29330]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:54.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:54 smithi155 ceph-mon[35815]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:56.550 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:56 smithi132 ceph-mon[29330]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:56.550 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:21:56.550 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:56.550 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:56 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:56.651 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:56 smithi155 ceph-mon[35815]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:56.651 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:21:56.651 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:56.651 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:56 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:58.484 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:58 smithi132 ceph-mon[29330]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:58.484 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:58 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:58.484 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:58 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:58.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.520+0000 7f7c81ffb640 1 -- 172.21.15.132:0/3400230994 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7c8c10ed50 con 0x7f7c780653a0 2024-06-13T01:21:58.519 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 1 on host 'smithi132' 2024-06-13T01:21:58.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c780653a0 msgr2=0x7f7c78067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c780653a0 0x7f7c78067850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c10cc90 tx=0x7f7c8800b040 comp rx=0 tx=0).stop 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 msgr2=0x7f7c8c10cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c10cee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c000c00 tx=0x7f7c7c009da0 comp rx=0 tx=0).stop 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 shutdown_connections 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c780653a0 0x7f7c78067850 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c8c0ff3e0 0x7f7c8c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 --2- 172.21.15.132:0/3400230994 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c8c0659a0 0x7f7c8c10cee0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 >> 172.21.15.132:0/3400230994 conn(0x7f7c8c0f6970 msgr2=0x7f7c8c0f6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 shutdown_connections 2024-06-13T01:21:58.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:21:58.522+0000 7f7c939fc640 1 -- 172.21.15.132:0/3400230994 wait complete. 2024-06-13T01:21:58.627 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:58 smithi155 ceph-mon[35815]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:21:58.627 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:58 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:58.627 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:58 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:59.154 DEBUG:teuthology.orchestra.run.smithi132:osd.1> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.1.service 2024-06-13T01:21:59.157 INFO:tasks.cephadm:Deploying osd.2 on smithi132 with /dev/vg_nvme/lv_2... 2024-06-13T01:21:59.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-13T01:21:59.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:59 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:59.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:59 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:59.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:21:59 smithi132 ceph-mon[29330]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-13T01:21:59.711 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:21:59 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1[54461]: 2024-06-13T01:21:59.454+0000 7f240d21e2c0 -1 osd.1 0 log_to_monitors true 2024-06-13T01:21:59.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:59 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:59.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:59 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:21:59.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:21:59 smithi155 ceph-mon[35815]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-13T01:22:00.729 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:22:00.852 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1[54461]: 2024-06-13T01:22:00.534+0000 7f2408990640 -1 osd.1 0 waiting for initial osdmap 2024-06-13T01:22:00.852 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1[54461]: 2024-06-13T01:22:00.539+0000 7f24047c9640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T01:22:00.852 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-mon[29330]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:22:00.852 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-mon[29330]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-13T01:22:00.852 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-mon[29330]: osdmap e12: 2 total, 1 up, 2 in 2024-06-13T01:22:00.852 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:00.852 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:00 smithi132 ceph-mon[29330]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-06-13T01:22:00.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:00 smithi155 ceph-mon[35815]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:22:00.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:00 smithi155 ceph-mon[35815]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-13T01:22:00.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:00 smithi155 ceph-mon[35815]: osdmap e12: 2 total, 1 up, 2 in 2024-06-13T01:22:00.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:00 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:00 smithi155 ceph-mon[35815]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-06-13T01:22:01.789 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:01 smithi132 ceph-mon[29330]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-06-13T01:22:01.789 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:01 smithi132 ceph-mon[29330]: osdmap e13: 2 total, 1 up, 2 in 2024-06-13T01:22:01.789 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:01 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:01.789 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:01 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:01.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:01 smithi155 ceph-mon[35815]: from='osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-06-13T01:22:01.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:01 smithi155 ceph-mon[35815]: osdmap e13: 2 total, 1 up, 2 in 2024-06-13T01:22:01.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:01 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:01.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:01 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: purged_snaps scrub starts 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: purged_snaps scrub ok 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] boot 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: osdmap e14: 2 total, 2 up, 2 in 2024-06-13T01:22:02.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:02 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:02.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: purged_snaps scrub starts 2024-06-13T01:22:02.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: purged_snaps scrub ok 2024-06-13T01:22:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-13T01:22:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: osd.1 [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] boot 2024-06-13T01:22:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: osdmap e14: 2 total, 2 up, 2 in 2024-06-13T01:22:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:02 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-13T01:22:03.218 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-13T01:22:03.307 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-13T01:22:03.322 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-06-13T01:22:03.322 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-06-13T01:22:03.322 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125995 s, 832 MB/s 2024-06-13T01:22:03.322 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-06-13T01:22:03.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:03 smithi155 ceph-mon[35815]: osdmap e15: 2 total, 2 up, 2 in 2024-06-13T01:22:03.892 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:03 smithi132 ceph-mon[29330]: osdmap e15: 2 total, 2 up, 2 in 2024-06-13T01:22:03.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_2 2024-06-13T01:22:04.362 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:22:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:04 smithi155 ceph-mon[35815]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:04.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:04.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:04.939 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:04 smithi132 ceph-mon[29330]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:04.939 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:04.939 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:04.939 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:06.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:06 smithi132 ceph-mon[29330]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:06.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.733+0000 7f86029bc640 1 -- 172.21.15.132:0/1055705894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc066230 msgr2=0x7f85fc0fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:06.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.733+0000 7f86029bc640 1 --2- 172.21.15.132:0/1055705894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc066230 0x7f85fc0fee90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0048d0 tx=0x7f85ec009da0 comp rx=0 tx=0).stop 2024-06-13T01:22:06.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- 172.21.15.132:0/1055705894 shutdown_connections 2024-06-13T01:22:06.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 --2- 172.21.15.132:0/1055705894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:06.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 --2- 172.21.15.132:0/1055705894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc066230 0x7f85fc0fee90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- 172.21.15.132:0/1055705894 >> 172.21.15.132:0/1055705894 conn(0x7f85fc0f6960 msgr2=0x7f85fc0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- 172.21.15.132:0/1055705894 shutdown_connections 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- 172.21.15.132:0/1055705894 wait complete. 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 Processor -- start 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- start start 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85fc066230 0x7f85fc10f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc10f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc10fc90 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.734+0000 7f86029bc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc1104b0 con 0x7f85fc066230 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc10f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc10f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44206/0 (socket says 172.21.15.132:44206) 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 -- 172.21.15.132:0/3533541888 learned_addr learned my addr 172.21.15.132:0/3533541888 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:22:06.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fbfff640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85fc066230 0x7f85fc10f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 -- 172.21.15.132:0/3533541888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85fc066230 msgr2=0x7f85fc10f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85fc066230 0x7f85fc10f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 -- 172.21.15.132:0/3533541888 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec005040 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85fb7fe640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc10f640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f85f000c790 tx=0x7f85f00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f000a7a0 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85f000add0 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc110740 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f00115e0 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.735+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc110ce0 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.736+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc0fa5e0 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.737+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f85f0012400 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.737+0000 7f85f8ff9640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f85e40653a0 0x7f85e4067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:06.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.737+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f85f0081b40 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.737+0000 7f85fbfff640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f85e40653a0 0x7f85e4067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:06.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.738+0000 7f85fbfff640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f85e40653a0 0x7f85e4067850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f85ec003080 tx=0x7f85ec035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:22:06.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.741+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f85f0050b60 con 0x7f85fc0ff3d0 2024-06-13T01:22:06.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:06 smithi155 ceph-mon[35815]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:06.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:06.920+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f85fc1110c0 con 0x7f85e40653a0 2024-06-13T01:22:07.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:07 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:22:07.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:07 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:22:07.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:07 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:07 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:22:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:07 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:22:07.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:07 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:08.683 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:08 smithi132 ceph-mon[29330]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:22:08.683 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:08 smithi132 ceph-mon[29330]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:08.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:08 smithi155 ceph-mon[35815]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:22:08.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:08 smithi155 ceph-mon[35815]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: Detected new or changed devices on smithi132 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: Adjusting osd_memory_target on smithi132 to 6432M 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:10.165 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:09 smithi132 ceph-mon[29330]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: Detected new or changed devices on smithi132 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: Adjusting osd_memory_target on smithi132 to 6432M 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:09 smithi155 ceph-mon[35815]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:11.139 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:10 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/550124323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe9868e-7649-4077-8868-d77818e0aedc"}]: dispatch 2024-06-13T01:22:11.140 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:10 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/550124323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe9868e-7649-4077-8868-d77818e0aedc"}]': finished 2024-06-13T01:22:11.140 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:10 smithi132 ceph-mon[29330]: osdmap e16: 3 total, 2 up, 3 in 2024-06-13T01:22:11.140 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:10 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:11.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:10 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/550124323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe9868e-7649-4077-8868-d77818e0aedc"}]: dispatch 2024-06-13T01:22:11.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:10 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/550124323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe9868e-7649-4077-8868-d77818e0aedc"}]': finished 2024-06-13T01:22:11.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:10 smithi155 ceph-mon[35815]: osdmap e16: 3 total, 2 up, 3 in 2024-06-13T01:22:11.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:10 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:12.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:11 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1158021897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:22:12.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:11 smithi132 ceph-mon[29330]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:12.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:11 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1158021897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:22:12.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:11 smithi155 ceph-mon[35815]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:14 smithi132 ceph-mon[29330]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:14 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:14 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:14 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:14 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:14 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:15.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:14 smithi155 ceph-mon[35815]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:15.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:14 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:15.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:14 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:15.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:14 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:15.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:14 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:15.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:14 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:16 smithi132 ceph-mon[29330]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:17.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:16 smithi155 ceph-mon[35815]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:17.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:17 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-13T01:22:17.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:17 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:18.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:17 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-13T01:22:18.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:17 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:19.086 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:18 smithi132 ceph-mon[29330]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:19.086 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:18 smithi132 ceph-mon[29330]: Deploying daemon osd.2 on smithi132 2024-06-13T01:22:19.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:18 smithi155 ceph-mon[35815]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:19.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:18 smithi155 ceph-mon[35815]: Deploying daemon osd.2 on smithi132 2024-06-13T01:22:20.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:20.142 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:21.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:20 smithi155 ceph-mon[35815]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:21.156 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:20 smithi132 ceph-mon[29330]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:23.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:22 smithi155 ceph-mon[35815]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:23.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:22 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:23.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:22 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:23.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:22 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:23.141 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:22 smithi132 ceph-mon[29330]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:23.141 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:22 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:23.141 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:22 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:23.141 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:22 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:25.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:24 smithi155 ceph-mon[35815]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:25.138 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:24 smithi132 ceph-mon[29330]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:26.800 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 2 on host 'smithi132' 2024-06-13T01:22:26.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.801+0000 7f85f8ff9640 1 -- 172.21.15.132:0/3533541888 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f85fc1110c0 con 0x7f85e40653a0 2024-06-13T01:22:26.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f85e40653a0 msgr2=0x7f85e4067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:26.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f85e40653a0 0x7f85e4067850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f85ec003080 tx=0x7f85ec035040 comp rx=0 tx=0).stop 2024-06-13T01:22:26.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 msgr2=0x7f85fc10f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:26.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc10f640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f85f000c790 tx=0x7f85f00043c0 comp rx=0 tx=0).stop 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 shutdown_connections 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f85e40653a0 0x7f85e4067850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85fc0ff3d0 0x7f85fc10f640 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 --2- 172.21.15.132:0/3533541888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85fc066230 0x7f85fc10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 >> 172.21.15.132:0/3533541888 conn(0x7f85fc0f6960 msgr2=0x7f85fc0faf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 shutdown_connections 2024-06-13T01:22:26.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:22:26.803+0000 7f86029bc640 1 -- 172.21.15.132:0/3533541888 wait complete. 2024-06-13T01:22:27.123 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:26 smithi132 ceph-mon[29330]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:27.124 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:26 smithi132 ceph-mon[29330]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T01:22:27.124 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:27.124 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:26 smithi155 ceph-mon[35815]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:26 smithi155 ceph-mon[35815]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-13T01:22:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:26 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:26 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:28.030 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:27 smithi132 ceph-mon[29330]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T01:22:28.030 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:27 smithi132 ceph-mon[29330]: osdmap e17: 3 total, 2 up, 3 in 2024-06-13T01:22:28.030 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:27 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:28.030 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:27 smithi132 ceph-mon[29330]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-06-13T01:22:28.055 DEBUG:teuthology.orchestra.run.smithi132:osd.2> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.2.service 2024-06-13T01:22:28.058 INFO:tasks.cephadm:Deploying osd.3 on smithi155 with /dev/vg_nvme/lv_4... 2024-06-13T01:22:28.058 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-13T01:22:28.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:27 smithi155 ceph-mon[35815]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-13T01:22:28.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:27 smithi155 ceph-mon[35815]: osdmap e17: 3 total, 2 up, 3 in 2024-06-13T01:22:28.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:27 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:28.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:27 smithi155 ceph-mon[35815]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-06-13T01:22:28.314 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:22:29.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:28 smithi155 ceph-mon[35815]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:28 smithi155 ceph-mon[35815]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-06-13T01:22:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:28 smithi155 ceph-mon[35815]: osdmap e18: 3 total, 2 up, 3 in 2024-06-13T01:22:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:28 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:28 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:28 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:29.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:28 smithi132 ceph-mon[29330]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:29.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:28 smithi132 ceph-mon[29330]: from='osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-06-13T01:22:29.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:28 smithi132 ceph-mon[29330]: osdmap e18: 3 total, 2 up, 3 in 2024-06-13T01:22:29.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:28 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:29.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:28 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:29.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:28 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:30.056 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: purged_snaps scrub starts 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: purged_snaps scrub ok 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] boot 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: osdmap e19: 3 total, 3 up, 3 in 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:30.057 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:29 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:30.129 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-13T01:22:30.143 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-06-13T01:22:30.143 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-06-13T01:22:30.143 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011297 s, 928 MB/s 2024-06-13T01:22:30.143 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: purged_snaps scrub starts 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: purged_snaps scrub ok 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: osd.2 [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] boot 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: osdmap e19: 3 total, 3 up, 3 in 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:30.174 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:29 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:30.765 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2024-06-13T01:22:31.016 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:22:31.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:30 smithi155 ceph-mon[35815]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:31.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:30 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-13T01:22:31.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:30 smithi155 ceph-mon[35815]: osdmap e20: 3 total, 3 up, 3 in 2024-06-13T01:22:31.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:30 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T01:22:31.134 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:30 smithi132 ceph-mon[29330]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-13T01:22:31.134 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:30 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-13T01:22:31.134 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:30 smithi132 ceph-mon[29330]: osdmap e20: 3 total, 3 up, 3 in 2024-06-13T01:22:31.135 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:30 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-13T01:22:31.429 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64021]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T01:22:31.430 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64021]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:31.430 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64021]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:31.430 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64021]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:31.710 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64027]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T01:22:31.710 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64027]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:31.711 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64027]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:31.711 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64027]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:32.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:31 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-13T01:22:32.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:31 smithi155 ceph-mon[35815]: osdmap e21: 3 total, 3 up, 3 in 2024-06-13T01:22:32.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:31 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:32.130 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-13T01:22:32.130 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:31 smithi132 ceph-mon[29330]: osdmap e21: 3 total, 3 up, 3 in 2024-06-13T01:22:32.130 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:31 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:32.130 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64328]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T01:22:32.131 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64328]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:32.131 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64328]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:32.131 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:31 smithi132 sudo[64328]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:32.431 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66715]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T01:22:32.432 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66715]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:32.432 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66715]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:32.432 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66715]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 -- 172.21.15.155:0/3281040376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fe270 msgr2=0x7f99780fa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 --2- 172.21.15.155:0/3281040376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fe270 0x7f99780fa3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f996c0048d0 tx=0x7f996c009da0 comp rx=0 tx=0).stop 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 -- 172.21.15.155:0/3281040376 shutdown_connections 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 --2- 172.21.15.155:0/3281040376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fe270 0x7f99780fa3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 --2- 172.21.15.155:0/3281040376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99780fd920 0x7f99780fdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 -- 172.21.15.155:0/3281040376 >> 172.21.15.155:0/3281040376 conn(0x7f99780f6120 msgr2=0x7f99780f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:22:32.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 -- 172.21.15.155:0/3281040376 shutdown_connections 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.497+0000 7f997fd09640 1 -- 172.21.15.155:0/3281040376 wait complete. 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997fd09640 1 Processor -- start 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997fd09640 1 -- start start 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997fd09640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 0x7f997810cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997fd09640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99780fe270 0x7f997810d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997fd09640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997810db00 con 0x7f99780fe270 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997fd09640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997810e320 con 0x7f99780fd920 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 0x7f997810cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 0x7f997810cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41054/0 (socket says 172.21.15.155:41054) 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 -- 172.21.15.155:0/2428233684 learned_addr learned my addr 172.21.15.155:0/2428233684 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:22:32.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997d2a4640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99780fe270 0x7f997810d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 -- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99780fe270 msgr2=0x7f997810d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99780fe270 0x7f997810d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 -- 172.21.15.155:0/2428233684 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c005040 con 0x7f99780fd920 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.498+0000 7f997daa5640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 0x7f997810cf70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f997000eb70 tx=0x7f997000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.499+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9970007c30 con 0x7f99780fd920 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.499+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f997000f040 con 0x7f99780fd920 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.499+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997810e550 con 0x7f99780fd920 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.499+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f997000cad0 con 0x7f99780fd920 2024-06-13T01:22:32.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.499+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997810eac0 con 0x7f99780fd920 2024-06-13T01:22:32.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.500+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9978105670 con 0x7f99780fd920 2024-06-13T01:22:32.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.500+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f997000cc30 con 0x7f99780fd920 2024-06-13T01:22:32.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.500+0000 7f99767fc640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9960065430 0x7f99600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:32.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.500+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f9970081aa0 con 0x7f99780fd920 2024-06-13T01:22:32.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.500+0000 7f997d2a4640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9960065430 0x7f99600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:32.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.501+0000 7f997d2a4640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9960065430 0x7f99600678e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:22:32.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.505+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9970050600 con 0x7f99780fd920 2024-06-13T01:22:32.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:32.674+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f997810ee40 con 0x7f9960065430 2024-06-13T01:22:32.692 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T01:22:32.692 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:32.692 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66726]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:32.693 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66726]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:32.693 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66729]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T01:22:32.955 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T01:22:32.956 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:32.956 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66732]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:32.956 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66732]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:32.956 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66729]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:32.956 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66729]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:32.956 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:22:32 smithi132 sudo[66729]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 sudo[66735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 sudo[66735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 sudo[66735]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 sudo[66735]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: mgrmap e19: smithi132.vjweoa(active, since 2m), standbys: smithi155.teuqcw 2024-06-13T01:22:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: osdmap e22: 3 total, 3 up, 3 in 2024-06-13T01:22:33.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:22:33.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:22:33.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:33.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:33 smithi132 ceph-mon[29330]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T01:22:33.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:33.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: mgrmap e19: smithi132.vjweoa(active, since 2m), standbys: smithi155.teuqcw 2024-06-13T01:22:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: osdmap e22: 3 total, 3 up, 3 in 2024-06-13T01:22:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:22:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:22:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 ceph-mon[35815]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T01:22:33.715 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38900]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-13T01:22:33.715 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38900]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:33.715 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38900]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:33.716 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38900]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:34.090 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38904]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-13T01:22:34.090 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38904]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-13T01:22:34.090 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38904]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-13T01:22:34.090 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:33 smithi155 sudo[38904]: pam_unix(sudo:session): session closed for user root 2024-06-13T01:22:34.350 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:22:34.350 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: Detected new or changed devices on smithi132 2024-06-13T01:22:34.350 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T01:22:34.350 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:34.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: Adjusting osd_memory_target on smithi132 to 4288M 2024-06-13T01:22:34.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:34.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:34.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: Detected new or changed devices on smithi132 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:34.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: Adjusting osd_memory_target on smithi132 to 4288M 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:34.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:35.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:35 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:35.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:35 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/1783008820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747fee8a-d594-4e25-afb9-fd1c0fe56083"}]: dispatch 2024-06-13T01:22:35.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:35 smithi132 ceph-mon[29330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747fee8a-d594-4e25-afb9-fd1c0fe56083"}]: dispatch 2024-06-13T01:22:35.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:35 smithi132 ceph-mon[29330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "747fee8a-d594-4e25-afb9-fd1c0fe56083"}]': finished 2024-06-13T01:22:35.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:35 smithi132 ceph-mon[29330]: osdmap e23: 4 total, 3 up, 4 in 2024-06-13T01:22:35.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:35 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:35.531 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:35 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:35.531 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:35 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/1783008820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747fee8a-d594-4e25-afb9-fd1c0fe56083"}]: dispatch 2024-06-13T01:22:35.531 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:35 smithi155 ceph-mon[35815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747fee8a-d594-4e25-afb9-fd1c0fe56083"}]: dispatch 2024-06-13T01:22:35.531 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:35 smithi155 ceph-mon[35815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "747fee8a-d594-4e25-afb9-fd1c0fe56083"}]': finished 2024-06-13T01:22:35.531 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:35 smithi155 ceph-mon[35815]: osdmap e23: 4 total, 3 up, 4 in 2024-06-13T01:22:35.532 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:35 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:36.460 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:36 smithi132 ceph-mon[29330]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:36.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:36 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/1792069387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:22:36.627 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:36 smithi155 ceph-mon[35815]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:36.628 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:36 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/1792069387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:22:39.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:38 smithi155 ceph-mon[35815]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:38 smithi132 ceph-mon[29330]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:39.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:39.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:39.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:39.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:41.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:40 smithi155 ceph-mon[35815]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-13T01:22:41.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:40 smithi132 ceph-mon[29330]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-13T01:22:43.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:42 smithi155 ceph-mon[35815]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-13T01:22:43.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:42 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-13T01:22:43.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:42 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:43.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:42 smithi132 ceph-mon[29330]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-13T01:22:43.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:42 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-13T01:22:43.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:42 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:43.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:43 smithi155 ceph-mon[35815]: Deploying daemon osd.3 on smithi155 2024-06-13T01:22:44.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:43 smithi132 ceph-mon[29330]: Deploying daemon osd.3 on smithi155 2024-06-13T01:22:44.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:44 smithi155 ceph-mon[35815]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-13T01:22:45.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:44 smithi132 ceph-mon[29330]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-13T01:22:46.971 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:46 smithi155 ceph-mon[35815]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-06-13T01:22:47.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:46 smithi132 ceph-mon[29330]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-06-13T01:22:48.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:47 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:48.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:47 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:48.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:47 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:48.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:47 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:48.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:47 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:48.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:47 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:49.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:48 smithi155 ceph-mon[35815]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-13T01:22:49.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:48 smithi132 ceph-mon[29330]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-13T01:22:50.125 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:50.125 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.126+0000 7f99767fc640 1 -- 172.21.15.155:0/2428233684 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f997810ee40 con 0x7f9960065430 2024-06-13T01:22:50.126 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 3 on host 'smithi155' 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.128+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9960065430 msgr2=0x7f99600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.128+0000 7f997fd09640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9960065430 0x7f99600678e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c035040 comp rx=0 tx=0).stop 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.128+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 msgr2=0x7f997810cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.128+0000 7f997fd09640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 0x7f997810cf70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f997000eb70 tx=0x7f997000b040 comp rx=0 tx=0).stop 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 shutdown_connections 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9960065430 0x7f99600678e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99780fe270 0x7f997810d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 --2- 172.21.15.155:0/2428233684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99780fd920 0x7f997810cf70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 >> 172.21.15.155:0/2428233684 conn(0x7f99780f6120 msgr2=0x7f99780f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:22:50.128 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 shutdown_connections 2024-06-13T01:22:50.129 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:50.129+0000 7f997fd09640 1 -- 172.21.15.155:0/2428233684 wait complete. 2024-06-13T01:22:50.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:22:51.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:50 smithi132 ceph-mon[29330]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-13T01:22:51.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:51.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:51.334 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:50 smithi155 ceph-mon[35815]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-13T01:22:51.334 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:51.334 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:51.361 DEBUG:teuthology.orchestra.run.smithi155:osd.3> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.3.service 2024-06-13T01:22:51.363 INFO:tasks.cephadm:Deploying osd.4 on smithi155 with /dev/vg_nvme/lv_3... 2024-06-13T01:22:51.364 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-13T01:22:51.678 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:22:52.103 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:51 smithi155 ceph-mon[35815]: from='osd.3 [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T01:22:52.103 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:51 smithi155 ceph-mon[35815]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T01:22:52.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:51 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:52.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:51 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:52.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:51 smithi132 ceph-mon[29330]: from='osd.3 [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T01:22:52.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:51 smithi132 ceph-mon[29330]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-13T01:22:52.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:51 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:52.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:51 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:53.014 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-mon[35815]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:53.014 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-mon[35815]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-13T01:22:53.014 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-mon[35815]: osdmap e24: 4 total, 3 up, 4 in 2024-06-13T01:22:53.014 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:53.014 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-mon[35815]: from='osd.3 [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:22:53.015 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-mon[35815]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:22:53.015 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3[41160]: 2024-06-13T01:22:52.903+0000 7f0d87eb5640 -1 osd.3 0 waiting for initial osdmap 2024-06-13T01:22:53.015 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:22:52 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3[41160]: 2024-06-13T01:22:52.909+0000 7f0d834ed640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T01:22:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:52 smithi132 ceph-mon[29330]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:52 smithi132 ceph-mon[29330]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-13T01:22:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:52 smithi132 ceph-mon[29330]: osdmap e24: 4 total, 3 up, 4 in 2024-06-13T01:22:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:52 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:52 smithi132 ceph-mon[29330]: from='osd.3 [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:22:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:52 smithi132 ceph-mon[29330]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:22:53.842 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-13T01:22:53.936 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-13T01:22:53.950 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-06-13T01:22:53.951 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-06-13T01:22:53.951 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123669 s, 848 MB/s 2024-06-13T01:22:53.951 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-06-13T01:22:54.157 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:53 smithi155 ceph-mon[35815]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-06-13T01:22:54.157 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:53 smithi155 ceph-mon[35815]: osdmap e25: 4 total, 3 up, 4 in 2024-06-13T01:22:54.157 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:53 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:54.157 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:53 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:54.157 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:53 smithi155 ceph-mon[35815]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:54.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:53 smithi132 ceph-mon[29330]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-06-13T01:22:54.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:53 smithi132 ceph-mon[29330]: osdmap e25: 4 total, 3 up, 4 in 2024-06-13T01:22:54.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:53 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:54.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:53 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:54.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:53 smithi132 ceph-mon[29330]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-13T01:22:54.794 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2024-06-13T01:22:55.079 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:54 smithi155 ceph-mon[35815]: purged_snaps scrub starts 2024-06-13T01:22:55.079 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:54 smithi155 ceph-mon[35815]: purged_snaps scrub ok 2024-06-13T01:22:55.079 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:55.079 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:54 smithi155 ceph-mon[35815]: osd.3 [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] boot 2024-06-13T01:22:55.079 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:54 smithi155 ceph-mon[35815]: osdmap e26: 4 total, 4 up, 4 in 2024-06-13T01:22:55.079 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:54 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:55.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:54 smithi132 ceph-mon[29330]: purged_snaps scrub starts 2024-06-13T01:22:55.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:54 smithi132 ceph-mon[29330]: purged_snaps scrub ok 2024-06-13T01:22:55.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:55.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:54 smithi132 ceph-mon[29330]: osd.3 [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] boot 2024-06-13T01:22:55.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:54 smithi132 ceph-mon[29330]: osdmap e26: 4 total, 4 up, 4 in 2024-06-13T01:22:55.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:54 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-13T01:22:55.322 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:22:56.045 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:55 smithi155 ceph-mon[35815]: osdmap e27: 4 total, 4 up, 4 in 2024-06-13T01:22:56.045 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:55 smithi155 ceph-mon[35815]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:22:56.045 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:55 smithi155 ceph-mon[35815]: osdmap e28: 4 total, 4 up, 4 in 2024-06-13T01:22:56.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:55 smithi132 ceph-mon[29330]: osdmap e27: 4 total, 4 up, 4 in 2024-06-13T01:22:56.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:55 smithi132 ceph-mon[29330]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:22:56.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:55 smithi132 ceph-mon[29330]: osdmap e28: 4 total, 4 up, 4 in 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 -- 172.21.15.155:0/1582655654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 msgr2=0x7f0fcc0fc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 --2- 172.21.15.155:0/1582655654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc0fc8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0048d0 tx=0x7f0fbc009da0 comp rx=0 tx=0).stop 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 -- 172.21.15.155:0/1582655654 shutdown_connections 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 --2- 172.21.15.155:0/1582655654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc0fc8d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 --2- 172.21.15.155:0/1582655654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fcc0fb190 0x7f0fcc0fb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 -- 172.21.15.155:0/1582655654 >> 172.21.15.155:0/1582655654 conn(0x7f0fcc0f6970 msgr2=0x7f0fcc0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 -- 172.21.15.155:0/1582655654 shutdown_connections 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 -- 172.21.15.155:0/1582655654 wait complete. 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 Processor -- start 2024-06-13T01:22:57.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.468+0000 7f0fd363e640 1 -- start start 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd363e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fcc0fb190 0x7f0fcc10f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd363e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc10f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd363e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10fc60 con 0x7f0fcc0fb190 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd363e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc110480 con 0x7f0fcc0fc460 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc10f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc10f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:48098/0 (socket says 172.21.15.155:48098) 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 -- 172.21.15.155:0/3310543271 learned_addr learned my addr 172.21.15.155:0/3310543271 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd13da640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fcc0fb190 0x7f0fcc10f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 -- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fcc0fb190 msgr2=0x7f0fcc10f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fcc0fb190 0x7f0fcc10f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 -- 172.21.15.155:0/3310543271 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc005040 con 0x7f0fcc0fc460 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd0bd9640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc10f610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc003080 tx=0x7f0fbc009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:22:57.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fbc038070 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc1106b0 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fbc032970 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.469+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fbc033900 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.470+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc110bf0 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.471+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc0fa5f0 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.472+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0fbc030020 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.472+0000 7f0fc1ffb640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0fb8065430 0x7f0fb80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.472+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f0fbc0a2190 con 0x7f0fcc0fc460 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.472+0000 7f0fd13da640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0fb8065430 0x7f0fb80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.473+0000 7f0fd13da640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0fb8065430 0x7f0fb80678e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8007fd0 tx=0x7f0fc8009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:22:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.476+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0fbc070a50 con 0x7f0fcc0fc460 2024-06-13T01:22:57.657 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:22:57.657+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fcc103d50 con 0x7f0fb8065430 2024-06-13T01:22:57.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: Detected new or changed devices on smithi155 2024-06-13T01:22:57.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: Adjusting osd_memory_target on smithi155 to 15936M 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:57 smithi155 ceph-mon[35815]: osdmap e29: 4 total, 4 up, 4 in 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: Detected new or changed devices on smithi155 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: Adjusting osd_memory_target on smithi155 to 15936M 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:22:57.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:57.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:22:57.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:22:57.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:57 smithi132 ceph-mon[29330]: osdmap e29: 4 total, 4 up, 4 in 2024-06-13T01:22:58.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:58 smithi132 ceph-mon[29330]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:22:58.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:58 smithi132 ceph-mon[29330]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:22:58.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:58 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:22:58.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:58 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:22:58.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:58 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:58.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:22:58 smithi132 ceph-mon[29330]: osdmap e30: 4 total, 4 up, 4 in 2024-06-13T01:22:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:58 smithi155 ceph-mon[35815]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:22:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:58 smithi155 ceph-mon[35815]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:22:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:58 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:22:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:58 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:22:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:58 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:22:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:22:58 smithi155 ceph-mon[35815]: osdmap e30: 4 total, 4 up, 4 in 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/1745635096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa586deb-7be2-4b28-a48c-8bb58513e492"}]: dispatch 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa586deb-7be2-4b28-a48c-8bb58513e492"}]: dispatch 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa586deb-7be2-4b28-a48c-8bb58513e492"}]': finished 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: osdmap e31: 5 total, 4 up, 5 in 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:00 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/772911400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:23:00.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-06-13T01:23:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/1745635096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa586deb-7be2-4b28-a48c-8bb58513e492"}]: dispatch 2024-06-13T01:23:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa586deb-7be2-4b28-a48c-8bb58513e492"}]: dispatch 2024-06-13T01:23:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa586deb-7be2-4b28-a48c-8bb58513e492"}]': finished 2024-06-13T01:23:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: osdmap e31: 5 total, 4 up, 5 in 2024-06-13T01:23:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:00 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/772911400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:23:02.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:02 smithi155 ceph-mon[35815]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-06-13T01:23:02.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:02 smithi132 ceph-mon[29330]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-06-13T01:23:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:04 smithi155 ceph-mon[35815]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-13T01:23:04.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:04.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:04 smithi132 ceph-mon[29330]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-13T01:23:04.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:06.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:06 smithi132 ceph-mon[29330]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-13T01:23:07.060 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:06 smithi155 ceph-mon[35815]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-13T01:23:07.821 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:07 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-13T01:23:07.821 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:07 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:07.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:07 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-13T01:23:07.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:07 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:08.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:08 smithi132 ceph-mon[29330]: Deploying daemon osd.4 on smithi155 2024-06-13T01:23:08.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:08 smithi132 ceph-mon[29330]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:08.761 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:08 smithi155 ceph-mon[35815]: Deploying daemon osd.4 on smithi155 2024-06-13T01:23:08.761 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:08 smithi155 ceph-mon[35815]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:10.926 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:10 smithi155 ceph-mon[35815]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:10.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:10 smithi132 ceph-mon[29330]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:12.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:12 smithi132 ceph-mon[29330]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:12.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:12 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:12.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:12 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:12.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:12 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:12.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:12 smithi155 ceph-mon[35815]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:12.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:12 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:12.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:12 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:12.968 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:12 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:14.685 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:14 smithi155 ceph-mon[35815]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:14 smithi132 ceph-mon[29330]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:15.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.633+0000 7f0fc1ffb640 1 -- 172.21.15.155:0/3310543271 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0fcc103d50 con 0x7f0fb8065430 2024-06-13T01:23:15.632 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 4 on host 'smithi155' 2024-06-13T01:23:15.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0fb8065430 msgr2=0x7f0fb80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:15.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0fb8065430 0x7f0fb80678e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8007fd0 tx=0x7f0fc8009040 comp rx=0 tx=0).stop 2024-06-13T01:23:15.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 msgr2=0x7f0fcc10f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:15.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc10f610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc003080 tx=0x7f0fbc009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:15.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 shutdown_connections 2024-06-13T01:23:15.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0fb8065430 0x7f0fb80678e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:15.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fcc0fc460 0x7f0fcc10f610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:15.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 --2- 172.21.15.155:0/3310543271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fcc0fb190 0x7f0fcc10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:15.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 >> 172.21.15.155:0/3310543271 conn(0x7f0fcc0f6970 msgr2=0x7f0fcc065270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:15.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 shutdown_connections 2024-06-13T01:23:15.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:15.635+0000 7f0fd363e640 1 -- 172.21.15.155:0/3310543271 wait complete. 2024-06-13T01:23:16.557 DEBUG:teuthology.orchestra.run.smithi155:osd.4> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.4.service 2024-06-13T01:23:16.560 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_2... 2024-06-13T01:23:16.560 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-13T01:23:16.859 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:23:16.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:16 smithi155 ceph-mon[35815]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:16.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:16 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:16.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:16 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:16.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:16 smithi155 ceph-mon[35815]: from='osd.4 [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T01:23:16.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:16 smithi155 ceph-mon[35815]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T01:23:16.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:16 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:16.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:16 smithi132 ceph-mon[29330]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:16 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:16 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:16 smithi132 ceph-mon[29330]: from='osd.4 [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T01:23:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:16 smithi132 ceph-mon[29330]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-13T01:23:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:16 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:17.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:17.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-mon[35815]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-13T01:23:17.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-mon[35815]: osdmap e32: 5 total, 4 up, 5 in 2024-06-13T01:23:17.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:17.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-mon[35815]: from='osd.4 [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:17.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-mon[35815]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:17.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:23:17.657+0000 7f62dae0a640 -1 osd.4 0 waiting for initial osdmap 2024-06-13T01:23:17.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:23:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:23:17.664+0000 7f62d3c1b640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T01:23:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:17 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:17 smithi132 ceph-mon[29330]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-13T01:23:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:17 smithi132 ceph-mon[29330]: osdmap e32: 5 total, 4 up, 5 in 2024-06-13T01:23:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:17 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:17 smithi132 ceph-mon[29330]: from='osd.4 [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:17 smithi132 ceph-mon[29330]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:18.769 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:18 smithi155 ceph-mon[35815]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:18.770 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:18 smithi155 ceph-mon[35815]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-06-13T01:23:18.770 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:18 smithi155 ceph-mon[35815]: osdmap e33: 5 total, 4 up, 5 in 2024-06-13T01:23:18.770 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:18 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:18.770 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:18 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:18.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:18 smithi132 ceph-mon[29330]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:18.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:18 smithi132 ceph-mon[29330]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-06-13T01:23:18.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:18 smithi132 ceph-mon[29330]: osdmap e33: 5 total, 4 up, 5 in 2024-06-13T01:23:18.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:18 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:18.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:18 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:19 smithi155 ceph-mon[35815]: purged_snaps scrub starts 2024-06-13T01:23:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:19 smithi155 ceph-mon[35815]: purged_snaps scrub ok 2024-06-13T01:23:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:19 smithi155 ceph-mon[35815]: osd.4 [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] boot 2024-06-13T01:23:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:19 smithi155 ceph-mon[35815]: osdmap e34: 5 total, 5 up, 5 in 2024-06-13T01:23:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:19 smithi132 ceph-mon[29330]: purged_snaps scrub starts 2024-06-13T01:23:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:19 smithi132 ceph-mon[29330]: purged_snaps scrub ok 2024-06-13T01:23:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:19 smithi132 ceph-mon[29330]: osd.4 [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] boot 2024-06-13T01:23:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:19 smithi132 ceph-mon[29330]: osdmap e34: 5 total, 5 up, 5 in 2024-06-13T01:23:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-13T01:23:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:20.667 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-13T01:23:20.757 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-13T01:23:20.771 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-06-13T01:23:20.771 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-06-13T01:23:20.771 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113227 s, 926 MB/s 2024-06-13T01:23:20.771 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-06-13T01:23:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: osdmap e35: 5 total, 5 up, 5 in 2024-06-13T01:23:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:20.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:20 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:21.061 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-13T01:23:21.061 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: osdmap e35: 5 total, 5 up, 5 in 2024-06-13T01:23:21.061 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:21.061 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:21.061 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:21.061 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:21.062 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:20 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:21.378 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2024-06-13T01:23:21.629 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:23:21.923 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:21 smithi155 ceph-mon[35815]: Detected new or changed devices on smithi155 2024-06-13T01:23:21.923 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:21 smithi155 ceph-mon[35815]: Adjusting osd_memory_target on smithi155 to 7968M 2024-06-13T01:23:21.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:21 smithi132 ceph-mon[29330]: Detected new or changed devices on smithi155 2024-06-13T01:23:21.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:21 smithi132 ceph-mon[29330]: Adjusting osd_memory_target on smithi155 to 7968M 2024-06-13T01:23:22.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:22 smithi132 ceph-mon[29330]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:23.017 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:22 smithi155 ceph-mon[35815]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:23.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 -- 172.21.15.155:0/2173961642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 msgr2=0x7f20100fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 --2- 172.21.15.155:0/2173961642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f20100fd9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80048d0 tx=0x7f1ff8009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 -- 172.21.15.155:0/2173961642 shutdown_connections 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 --2- 172.21.15.155:0/2173961642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f20100fd9c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 --2- 172.21.15.155:0/2173961642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20100fc280 0x7f20100fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 -- 172.21.15.155:0/2173961642 >> 172.21.15.155:0/2173961642 conn(0x7f20100f7a40 msgr2=0x7f20100f9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 -- 172.21.15.155:0/2173961642 shutdown_connections 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 -- 172.21.15.155:0/2173961642 wait complete. 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 Processor -- start 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.229+0000 7f20168df640 1 -- start start 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f20168df640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20100fc280 0x7f201010cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:23.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f20168df640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f201010d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f20168df640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201010da30 con 0x7f20100fc280 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f20168df640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201010e250 con 0x7f20100fd550 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f201010d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f201010d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:35490/0 (socket says 172.21.15.155:35490) 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 -- 172.21.15.155:0/831076738 learned_addr learned my addr 172.21.15.155:0/831076738 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200ffff640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20100fc280 0x7f201010cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 -- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20100fc280 msgr2=0x7f201010cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20100fc280 0x7f201010cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 -- 172.21.15.155:0/831076738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8005040 con 0x7f20100fd550 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200f7fe640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f201010d3e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009f80 tx=0x7f1ff8009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff8038070 con 0x7f20100fd550 2024-06-13T01:23:23.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201010e480 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ff8032970 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff8033860 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.230+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201010e9c0 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.231+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20100fb6e0 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.232+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1ff8030020 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.232+0000 7f200cff9640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ffc065430 0x7f1ffc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.232+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f1ff80a2190 con 0x7f20100fd550 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.232+0000 7f200ffff640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ffc065430 0x7f1ffc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.233+0000 7f200ffff640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ffc065430 0x7f1ffc0678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2000004ed0 tx=0x7f200000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:23.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.236+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1ff80708a0 con 0x7f20100fd550 2024-06-13T01:23:23.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:23.409+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f201010ecf0 con 0x7f1ffc065430 2024-06-13T01:23:23.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:23:23.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:23:23.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:23 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:23.987 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-13T01:23:23.987 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-13T01:23:23.987 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:23 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:24.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:24 smithi132 ceph-mon[29330]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:24.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:24 smithi132 ceph-mon[29330]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:23:25.001 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:24 smithi155 ceph-mon[35815]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:25.002 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:24 smithi155 ceph-mon[35815]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/3742730607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26458ee-737c-4e2b-a627-793ba7098bbb"}]: dispatch 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26458ee-737c-4e2b-a627-793ba7098bbb"}]: dispatch 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b26458ee-737c-4e2b-a627-793ba7098bbb"}]': finished 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: osdmap e36: 6 total, 5 up, 6 in 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:26 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/2784551531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:23:26.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/3742730607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26458ee-737c-4e2b-a627-793ba7098bbb"}]: dispatch 2024-06-13T01:23:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26458ee-737c-4e2b-a627-793ba7098bbb"}]: dispatch 2024-06-13T01:23:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b26458ee-737c-4e2b-a627-793ba7098bbb"}]': finished 2024-06-13T01:23:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: osdmap e36: 6 total, 5 up, 6 in 2024-06-13T01:23:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:26 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/2784551531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-13T01:23:28.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:28 smithi132 ceph-mon[29330]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:29.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:28 smithi155 ceph-mon[35815]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:30.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:30 smithi155 ceph-mon[35815]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:30.960 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:30 smithi132 ceph-mon[29330]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:32.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:32 smithi132 ceph-mon[29330]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:32 smithi155 ceph-mon[35815]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:33.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-13T01:23:33.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:33 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:34.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-13T01:23:34.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:33 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:34.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:34 smithi155 ceph-mon[35815]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:34.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:34 smithi155 ceph-mon[35815]: Deploying daemon osd.5 on smithi155 2024-06-13T01:23:34.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:35.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:34 smithi132 ceph-mon[29330]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:35.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:34 smithi132 ceph-mon[29330]: Deploying daemon osd.5 on smithi155 2024-06-13T01:23:35.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:36.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:36 smithi155 ceph-mon[35815]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:37.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:36 smithi132 ceph-mon[29330]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:39.022 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:38 smithi155 ceph-mon[35815]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:39.022 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:39.022 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:39.022 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:38 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:38 smithi132 ceph-mon[29330]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:38 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:40.995 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:40 smithi155 ceph-mon[35815]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:41.210 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:40 smithi132 ceph-mon[29330]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:41.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.518+0000 7f200cff9640 1 -- 172.21.15.155:0/831076738 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f201010ecf0 con 0x7f1ffc065430 2024-06-13T01:23:41.517 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2024-06-13T01:23:41.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ffc065430 msgr2=0x7f1ffc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:41.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ffc065430 0x7f1ffc0678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2000004ed0 tx=0x7f200000b040 comp rx=0 tx=0).stop 2024-06-13T01:23:41.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 msgr2=0x7f201010d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:41.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f201010d3e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009f80 tx=0x7f1ff8009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:41.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 shutdown_connections 2024-06-13T01:23:41.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ffc065430 0x7f1ffc0678e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:41.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20100fd550 0x7f201010d3e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:41.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 --2- 172.21.15.155:0/831076738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20100fc280 0x7f201010cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:41.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 >> 172.21.15.155:0/831076738 conn(0x7f20100f7a40 msgr2=0x7f20100f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:41.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 shutdown_connections 2024-06-13T01:23:41.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:41.520+0000 7f20168df640 1 -- 172.21.15.155:0/831076738 wait complete. 2024-06-13T01:23:42.495 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.5.service 2024-06-13T01:23:42.497 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-13T01:23:42.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd stat -f json 2024-06-13T01:23:42.759 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:42 smithi155 ceph-mon[35815]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:42.759 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:42 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:42.759 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:42 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:42.759 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:42 smithi155 ceph-mon[35815]: from='osd.5 [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T01:23:42.759 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:42 smithi155 ceph-mon[35815]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T01:23:42.761 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:23:42.883 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:42 smithi132 ceph-mon[29330]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:42.883 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:42 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:42.883 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:42 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:42.883 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:42 smithi132 ceph-mon[29330]: from='osd.5 [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T01:23:42.884 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:42 smithi132 ceph-mon[29330]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-13T01:23:43.925 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-13T01:23:43.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: osdmap e37: 6 total, 5 up, 6 in 2024-06-13T01:23:43.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:43.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: from='osd.5 [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:43.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:43.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:43.926 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:43 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: osdmap e37: 6 total, 5 up, 6 in 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: from='osd.5 [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:44.129 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:23:43.736+0000 7f63e742c640 -1 osd.5 0 waiting for initial osdmap 2024-06-13T01:23:44.129 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:23:43 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:23:43.744+0000 7f63e025f640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-13T01:23:44.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/3931972551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fb050 msgr2=0x7f3c24101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:44.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/3931972551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fb050 0x7f3c24101880 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c180048d0 tx=0x7f3c18009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/3931972551 shutdown_connections 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/3931972551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fb050 0x7f3c24101880 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/3931972551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/3931972551 >> 172.21.15.132:0/3931972551 conn(0x7f3c240f60e0 msgr2=0x7f3c240f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/3931972551 shutdown_connections 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/3931972551 wait complete. 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.407+0000 7f3c2a9c6640 1 Processor -- start 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 -- start start 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c240fb050 0x7f3c240fb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c240fbd00 con 0x7f3c240fa6e0 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c240fbe40 con 0x7f3c240fb050 2024-06-13T01:23:44.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46474/0 (socket says 172.21.15.132:46474) 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 -- 172.21.15.132:0/462866788 learned_addr learned my addr 172.21.15.132:0/462866788 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c237fe640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c240fb050 0x7f3c240fb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 -- 172.21.15.132:0/462866788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c240fb050 msgr2=0x7f3c240fb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c240fb050 0x7f3c240fb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 -- 172.21.15.132:0/462866788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c18005040 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c23fff640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fd1f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400b580 tx=0x7f3c140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c1400f570 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c1400fba0 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c240fc120 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c14010dd0 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.408+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c24114fc0 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.409+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c240f9da0 con 0x7f3c240fa6e0 2024-06-13T01:23:44.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.410+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3c1400f6d0 con 0x7f3c240fa6e0 2024-06-13T01:23:44.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.410+0000 7f3c20ff9640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3c0c065430 0x7f3c0c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:44.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.410+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f3c14082300 con 0x7f3c240fa6e0 2024-06-13T01:23:44.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.410+0000 7f3c237fe640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3c0c065430 0x7f3c0c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:44.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.411+0000 7f3c237fe640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3c0c065430 0x7f3c0c0678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009f80 tx=0x7f3c18035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:44.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.415+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3c14050940 con 0x7f3c240fa6e0 2024-06-13T01:23:44.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.579+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3c24104ed0 con 0x7f3c240fa6e0 2024-06-13T01:23:44.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.579+0000 7f3c20ff9640 1 -- 172.21.15.132:0/462866788 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3c14053f60 con 0x7f3c240fa6e0 2024-06-13T01:23:44.579 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:23:44.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3c0c065430 msgr2=0x7f3c0c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:44.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3c0c065430 0x7f3c0c0678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009f80 tx=0x7f3c18035040 comp rx=0 tx=0).stop 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 msgr2=0x7f3c240fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fd1f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400b580 tx=0x7f3c140091a0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 shutdown_connections 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3c0c065430 0x7f3c0c0678e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c240fb050 0x7f3c240fb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 --2- 172.21.15.132:0/462866788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c240fa6e0 0x7f3c240fd1f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 >> 172.21.15.132:0/462866788 conn(0x7f3c240f60e0 msgr2=0x7f3c240ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 shutdown_connections 2024-06-13T01:23:44.581 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:44.581+0000 7f3c2a9c6640 1 -- 172.21.15.132:0/462866788 wait complete. 2024-06-13T01:23:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:44 smithi155 ceph-mon[35815]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:44 smithi155 ceph-mon[35815]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-06-13T01:23:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:44 smithi155 ceph-mon[35815]: osdmap e38: 6 total, 5 up, 6 in 2024-06-13T01:23:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:44 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:44 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:44 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/462866788' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T01:23:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:44 smithi132 ceph-mon[29330]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:44 smithi132 ceph-mon[29330]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-06-13T01:23:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:44 smithi132 ceph-mon[29330]: osdmap e38: 6 total, 5 up, 6 in 2024-06-13T01:23:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:44 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:44 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:44 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/462866788' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T01:23:45.173 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1718241798,"num_in_osds":6,"osd_in_since":1718241805,"num_remapped_pgs":0} 2024-06-13T01:23:45.743 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:45 smithi155 ceph-mon[35815]: purged_snaps scrub starts 2024-06-13T01:23:45.744 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:45 smithi155 ceph-mon[35815]: purged_snaps scrub ok 2024-06-13T01:23:45.744 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:45 smithi155 ceph-mon[35815]: osd.5 [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] boot 2024-06-13T01:23:45.744 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:45 smithi155 ceph-mon[35815]: osdmap e39: 6 total, 6 up, 6 in 2024-06-13T01:23:45.744 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:45 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:46.174 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd stat -f json 2024-06-13T01:23:46.203 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:45 smithi132 ceph-mon[29330]: purged_snaps scrub starts 2024-06-13T01:23:46.203 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:45 smithi132 ceph-mon[29330]: purged_snaps scrub ok 2024-06-13T01:23:46.203 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:45 smithi132 ceph-mon[29330]: osd.5 [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] boot 2024-06-13T01:23:46.203 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:45 smithi132 ceph-mon[29330]: osdmap e39: 6 total, 6 up, 6 in 2024-06-13T01:23:46.203 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:45 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-13T01:23:46.437 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:23:46.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:46 smithi132 ceph-mon[29330]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:46.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:46 smithi132 ceph-mon[29330]: osdmap e40: 6 total, 6 up, 6 in 2024-06-13T01:23:47.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:46 smithi155 ceph-mon[35815]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-13T01:23:47.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:46 smithi155 ceph-mon[35815]: osdmap e40: 6 total, 6 up, 6 in 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 -- 172.21.15.132:0/74117872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 msgr2=0x7f58940fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 --2- 172.21.15.132:0/74117872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f58940fc690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f58840048d0 tx=0x7f5884009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 -- 172.21.15.132:0/74117872 shutdown_connections 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 --2- 172.21.15.132:0/74117872 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 0x7f58940fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 --2- 172.21.15.132:0/74117872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f58940fc690 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 -- 172.21.15.132:0/74117872 >> 172.21.15.132:0/74117872 conn(0x7f58940f7a40 msgr2=0x7f58940f9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 -- 172.21.15.132:0/74117872 shutdown_connections 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.030+0000 7f589979e640 1 -- 172.21.15.132:0/74117872 wait complete. 2024-06-13T01:23:48.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f589979e640 1 Processor -- start 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f589979e640 1 -- start start 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f589979e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f589410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f589979e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 0x7f589410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f589979e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410da40 con 0x7f58940fc280 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f589979e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410e260 con 0x7f58940fd550 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f5892ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f589410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f5892ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f589410ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46498/0 (socket says 172.21.15.132:46498) 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f5892ffd640 1 -- 172.21.15.132:0/3811662406 learned_addr learned my addr 172.21.15.132:0/3811662406 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f58927fc640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 0x7f589410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f5892ffd640 1 -- 172.21.15.132:0/3811662406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 msgr2=0x7f589410d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f5892ffd640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 0x7f589410d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f5892ffd640 1 -- 172.21.15.132:0/3811662406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884005040 con 0x7f58940fc280 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.031+0000 7f58927fc640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 0x7f589410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.032+0000 7f5892ffd640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f589410ceb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5884009f80 tx=0x7f5884009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.032+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5884038070 con 0x7f58940fc280 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.032+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5884032970 con 0x7f58940fc280 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.032+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589410e490 con 0x7f58940fc280 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.032+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5884033860 con 0x7f58940fc280 2024-06-13T01:23:48.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.032+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589410e9d0 con 0x7f58940fc280 2024-06-13T01:23:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.033+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58940fb6e0 con 0x7f58940fc280 2024-06-13T01:23:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.033+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5884030020 con 0x7f58940fc280 2024-06-13T01:23:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.034+0000 7f587b7fe640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f587c065430 0x7f587c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.034+0000 7f58927fc640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f587c065430 0x7f587c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.034+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58840a2340 con 0x7f58940fc280 2024-06-13T01:23:48.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.034+0000 7f58927fc640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f587c065430 0x7f587c0678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5888007fd0 tx=0x7f5888009e90 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:48.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.038+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5884070870 con 0x7f58940fc280 2024-06-13T01:23:48.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.204+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f589404b480 con 0x7f58940fc280 2024-06-13T01:23:48.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.204+0000 7f587b7fe640 1 -- 172.21.15.132:0/3811662406 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f5884041090 con 0x7f58940fc280 2024-06-13T01:23:48.206 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: Detected new or changed devices on smithi155 2024-06-13T01:23:48.206 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.206 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.206 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:48.206 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:48.206 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: Adjusting osd_memory_target on smithi155 to 5312M 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.207 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:48 smithi132 ceph-mon[29330]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:48.207 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:23:48.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f587c065430 msgr2=0x7f587c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:48.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f587c065430 0x7f587c0678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5888007fd0 tx=0x7f5888009e90 comp rx=0 tx=0).stop 2024-06-13T01:23:48.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 msgr2=0x7f589410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:48.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f589410ceb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5884009f80 tx=0x7f5884009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 shutdown_connections 2024-06-13T01:23:48.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f587c065430 0x7f587c0678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58940fd550 0x7f589410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 --2- 172.21.15.132:0/3811662406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58940fc280 0x7f589410ceb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:48.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.206+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 >> 172.21.15.132:0/3811662406 conn(0x7f58940f7a40 msgr2=0x7f58940f9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:48.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.207+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 shutdown_connections 2024-06-13T01:23:48.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:48.207+0000 7f589979e640 1 -- 172.21.15.132:0/3811662406 wait complete. 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: Detected new or changed devices on smithi155 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: Adjusting osd_memory_target on smithi155 to 5312M 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:23:48.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:48 smithi155 ceph-mon[35815]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:48.789 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1718241824,"num_in_osds":6,"osd_in_since":1718241805,"num_remapped_pgs":0} 2024-06-13T01:23:48.789 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T01:23:49.080 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:23:49.202 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:49 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3811662406' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T01:23:49.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:49 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3811662406' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-13T01:23:50.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:50.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:50 smithi132 ceph-mon[29330]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:50.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:23:50.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:50 smithi155 ceph-mon[35815]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 -- 172.21.15.132:0/1185597948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc1022a0 msgr2=0x7fb3dc102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 --2- 172.21.15.132:0/1185597948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc1022a0 0x7fb3dc102670 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009c50 tx=0x7fb3cc004d10 comp rx=0 tx=0).stop 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 -- 172.21.15.132:0/1185597948 shutdown_connections 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 --2- 172.21.15.132:0/1185597948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3dc102c40 0x7fb3dc0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 --2- 172.21.15.132:0/1185597948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc1022a0 0x7fb3dc102670 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 -- 172.21.15.132:0/1185597948 >> 172.21.15.132:0/1185597948 conn(0x7fb3dc0f7e50 msgr2=0x7fb3dc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:50.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 -- 172.21.15.132:0/1185597948 shutdown_connections 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.762+0000 7fb3e1407640 1 -- 172.21.15.132:0/1185597948 wait complete. 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3e1407640 1 Processor -- start 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3e1407640 1 -- start start 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3e1407640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3dc1022a0 0x7fb3dc113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3e1407640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 0x7fb3dc113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3e1407640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc113b40 con 0x7fb3dc102c40 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3e1407640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc10cf80 con 0x7fb3dc1022a0 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3da7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 0x7fb3dc113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3da7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 0x7fb3dc113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46520/0 (socket says 172.21.15.132:46520) 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3da7fc640 1 -- 172.21.15.132:0/3668172652 learned_addr learned my addr 172.21.15.132:0/3668172652 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3da7fc640 1 -- 172.21.15.132:0/3668172652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3dc1022a0 msgr2=0x7fb3dc113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:50.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3da7fc640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3dc1022a0 0x7fb3dc113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.763+0000 7fb3da7fc640 1 -- 172.21.15.132:0/3668172652 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0098b0 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.764+0000 7fb3da7fc640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 0x7fb3dc113540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000c790 tx=0x7fb3d00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.764+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d000a820 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.764+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3d000ae50 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.764+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc10d260 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.764+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d00115e0 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.764+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc10d780 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.765+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc0fda10 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.765+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb3d0012400 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.765+0000 7fb3c37fe640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb3c4065430 0x7fb3c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.766+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb3d0082690 con 0x7fb3dc102c40 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.766+0000 7fb3daffd640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb3c4065430 0x7fb3c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:50.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.766+0000 7fb3daffd640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb3c4065430 0x7fb3c40678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc004750 tx=0x7fb3cc004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:50.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.770+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb3d0050bc0 con 0x7fb3dc102c40 2024-06-13T01:23:50.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.935+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3dc0fba90 con 0x7fb3dc102c40 2024-06-13T01:23:50.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.936+0000 7fb3c37fe640 1 -- 172.21.15.132:0/3668172652 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11123 (secure 0 0 0) 0x7fb3d00541e0 con 0x7fb3dc102c40 2024-06-13T01:23:50.935 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:23:50.936 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":40,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","created":"2024-06-13T01:18:26.791525+0000","modified":"2024-06-13T01:23:45.740129+0000","last_up_change":"2024-06-13T01:23:44.734688+0000","last_in_change":"2024-06-13T01:23:25.927571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T01:22:29.617177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c3479cf5-50de-49b8-8219-8373f5329897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6803","nonce":3154009542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6805","nonce":3154009542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6809","nonce":3154009542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6807","nonce":3154009542}]},"public_addr":"172.21.15.132:6803/3154009542","cluster_addr":"172.21.15.132:6805/3154009542","heartbeat_back_addr":"172.21.15.132:6809/3154009542","heartbeat_front_addr":"172.21.15.132:6807/3154009542","state":["exists","up"]},{"osd":1,"uuid":"77f55f5e-4836-40db-9b9c-712be934facf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6811","nonce":2195816014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6813","nonce":2195816014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6817","nonce":2195816014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6815","nonce":2195816014}]},"public_addr":"172.21.15.132:6811/2195816014","cluster_addr":"172.21.15.132:6813/2195816014","heartbeat_back_addr":"172.21.15.132:6817/2195816014","heartbeat_front_addr":"172.21.15.132:6815/2195816014","state":["exists","up"]},{"osd":2,"uuid":"6fe9868e-7649-4077-8868-d77818e0aedc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6819","nonce":2883035986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6821","nonce":2883035986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6825","nonce":2883035986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6823","nonce":2883035986}]},"public_addr":"172.21.15.132:6819/2883035986","cluster_addr":"172.21.15.132:6821/2883035986","heartbeat_back_addr":"172.21.15.132:6825/2883035986","heartbeat_front_addr":"172.21.15.132:6823/2883035986","state":["exists","up"]},{"osd":3,"uuid":"747fee8a-d594-4e25-afb9-fd1c0fe56083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6801","nonce":920787201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6803","nonce":920787201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6807","nonce":920787201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6805","nonce":920787201}]},"public_addr":"172.21.15.155:6801/920787201","cluster_addr":"172.21.15.155:6803/920787201","heartbeat_back_addr":"172.21.15.155:6807/920787201","heartbeat_front_addr":"172.21.15.155:6805/920787201","state":["exists","up"]},{"osd":4,"uuid":"aa586deb-7be2-4b28-a48c-8bb58513e492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6809","nonce":3401072403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6811","nonce":3401072403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6815","nonce":3401072403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6813","nonce":3401072403}]},"public_addr":"172.21.15.155:6809/3401072403","cluster_addr":"172.21.15.155:6811/3401072403","heartbeat_back_addr":"172.21.15.155:6815/3401072403","heartbeat_front_addr":"172.21.15.155:6813/3401072403","state":["exists","up"]},{"osd":5,"uuid":"b26458ee-737c-4e2b-a627-793ba7098bbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6817","nonce":410629247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6819","nonce":410629247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6823","nonce":410629247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6821","nonce":410629247}]},"public_addr":"172.21.15.155:6817/410629247","cluster_addr":"172.21.15.155:6819/410629247","heartbeat_back_addr":"172.21.15.155:6823/410629247","heartbeat_front_addr":"172.21.15.155:6821/410629247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:21:32.386358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:00.506581+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:27.579430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:51.894977+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:23:17.195060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:23:43.496687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.132:6800/3829496564":"2024-06-14T01:20:19.303360+0000","172.21.15.132:6801/3829496564":"2024-06-14T01:20:19.303360+0000","172.21.15.132:0/1381082418":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6800/1802099589":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6801/1802099589":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6801/2668291289":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/2054197342":"2024-06-14T01:20:19.303360+0000","172.21.15.132:0/2808306670":"2024-06-14T01:18:48.373793+0000","172.21.15.132:0/331608642":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/2828217670":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/3751223226":"2024-06-14T01:18:48.373793+0000","172.21.15.132:0/4280057465":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/1942975292":"2024-06-14T01:20:19.303360+0000","172.21.15.132:6800/2668291289":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/3189205445":"2024-06-14T01:20:19.303360+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb3c4065430 msgr2=0x7fb3c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb3c4065430 0x7fb3c40678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc004750 tx=0x7fb3cc004d10 comp rx=0 tx=0).stop 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 msgr2=0x7fb3dc113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 0x7fb3dc113540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000c790 tx=0x7fb3d00043c0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 shutdown_connections 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb3c4065430 0x7fb3c40678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3dc102c40 0x7fb3dc113540 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 --2- 172.21.15.132:0/3668172652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3dc1022a0 0x7fb3dc113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:50.938 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.938+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 >> 172.21.15.132:0/3668172652 conn(0x7fb3dc0f7e50 msgr2=0x7fb3dc0f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:50.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.939+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 shutdown_connections 2024-06-13T01:23:50.939 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:50.939+0000 7fb3e1407640 1 -- 172.21.15.132:0/3668172652 wait complete. 2024-06-13T01:23:51.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:51 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3668172652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T01:23:51.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:51 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3668172652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T01:23:51.723 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-13T01:22:29.617177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-13T01:23:51.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-13T01:23:51.979 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:23:52.377 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:52 smithi155 ceph-mon[35815]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:52.400 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:52 smithi132 ceph-mon[29330]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.581+0000 7f43a163f640 1 -- 172.21.15.132:0/3005944123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 msgr2=0x7f439c0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.581+0000 7f43a163f640 1 --2- 172.21.15.132:0/3005944123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c0fb1e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f438c0048d0 tx=0x7f438c009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.581+0000 7f43a163f640 1 -- 172.21.15.132:0/3005944123 shutdown_connections 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.581+0000 7f43a163f640 1 --2- 172.21.15.132:0/3005944123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c0fb1e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.581+0000 7f43a163f640 1 --2- 172.21.15.132:0/3005944123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f439c1011b0 0x7f439c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.581+0000 7f43a163f640 1 -- 172.21.15.132:0/3005944123 >> 172.21.15.132:0/3005944123 conn(0x7f439c0f6d60 msgr2=0x7f439c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:53.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 -- 172.21.15.132:0/3005944123 shutdown_connections 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 -- 172.21.15.132:0/3005944123 wait complete. 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 Processor -- start 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 -- start start 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f439c1011b0 0x7f439c113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c113b70 con 0x7f439c101b50 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.582+0000 7f43a163f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c113ce0 con 0x7f439c1011b0 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44244/0 (socket says 172.21.15.132:44244) 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 -- 172.21.15.132:0/55941224 learned_addr learned my addr 172.21.15.132:0/55941224 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f439affd640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f439c1011b0 0x7f439c113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 -- 172.21.15.132:0/55941224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f439c1011b0 msgr2=0x7f439c113030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:53.583 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f439c1011b0 0x7f439c113030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 -- 172.21.15.132:0/55941224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c005040 con 0x7f439c101b50 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43927fc640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c113570 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f438c0043d0 tx=0x7f438c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f438c038070 con 0x7f439c101b50 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f438c032970 con 0x7f439c101b50 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c10d100 con 0x7f439c101b50 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f438c03da60 con 0x7f439c101b50 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.583+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c10d640 con 0x7f439c101b50 2024-06-13T01:23:53.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.584+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c0fc920 con 0x7f439c101b50 2024-06-13T01:23:53.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.585+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f438c030020 con 0x7f439c101b50 2024-06-13T01:23:53.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.585+0000 7f4393fff640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4388065430 0x7f43880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:53.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.585+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f438c0a25c0 con 0x7f439c101b50 2024-06-13T01:23:53.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.585+0000 7f439affd640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4388065430 0x7f43880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:53.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.585+0000 7f439affd640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4388065430 0x7f43880678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4384004ed0 tx=0x7f438400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:53.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.589+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f438c070a70 con 0x7f439c101b50 2024-06-13T01:23:53.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.754+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f439c04b480 con 0x7f439c101b50 2024-06-13T01:23:53.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.754+0000 7f4393fff640 1 -- 172.21.15.132:0/55941224 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f438c074090 con 0x7f439c101b50 2024-06-13T01:23:53.754 INFO:teuthology.orchestra.run.smithi132.stdout:pg_num: 1 2024-06-13T01:23:53.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4388065430 msgr2=0x7f43880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:53.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4388065430 0x7f43880678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4384004ed0 tx=0x7f438400b040 comp rx=0 tx=0).stop 2024-06-13T01:23:53.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 msgr2=0x7f439c113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c113570 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f438c0043d0 tx=0x7f438c009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 shutdown_connections 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4388065430 0x7f43880678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f439c101b50 0x7f439c113570 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 --2- 172.21.15.132:0/55941224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f439c1011b0 0x7f439c113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 >> 172.21.15.132:0/55941224 conn(0x7f439c0f6d60 msgr2=0x7f439c0657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 shutdown_connections 2024-06-13T01:23:53.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:53.756+0000 7f43a163f640 1 -- 172.21.15.132:0/55941224 wait complete. 2024-06-13T01:23:54.520 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:54 smithi132 ceph-mon[29330]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:54.520 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:54 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/55941224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-13T01:23:54.521 INFO:tasks.cephadm:Setting up client nodes... 2024-06-13T01:23:54.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-13T01:23:54.776 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:23:54.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:54 smithi155 ceph-mon[35815]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:54.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:54 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/55941224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-13T01:23:56.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 -- 172.21.15.132:0/144598385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 msgr2=0x7f15700fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:56.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 --2- 172.21.15.132:0/144598385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700fc6c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f15600048d0 tx=0x7f1560009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 -- 172.21.15.132:0/144598385 shutdown_connections 2024-06-13T01:23:56.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 --2- 172.21.15.132:0/144598385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15700fd580 0x7f15700fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 --2- 172.21.15.132:0/144598385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700fc6c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 -- 172.21.15.132:0/144598385 >> 172.21.15.132:0/144598385 conn(0x7f15700f7a70 msgr2=0x7f15700f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 -- 172.21.15.132:0/144598385 shutdown_connections 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 -- 172.21.15.132:0/144598385 wait complete. 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.376+0000 7f1574a90640 1 Processor -- start 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1574a90640 1 -- start start 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1574a90640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1574a90640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15700fd580 0x7f157006f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1574a90640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157006fd50 con 0x7f15700fc2b0 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1574a90640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157006fe90 con 0x7f15700fd580 2024-06-13T01:23:56.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44260/0 (socket says 172.21.15.132:44260) 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 -- 172.21.15.132:0/1092596419 learned_addr learned my addr 172.21.15.132:0/1092596419 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1567fff640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15700fd580 0x7f157006f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 -- 172.21.15.132:0/1092596419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15700fd580 msgr2=0x7f157006f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15700fd580 0x7f157006f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 -- 172.21.15.132:0/1092596419 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560005040 con 0x7f15700fc2b0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f156e59c640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700712d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1560009f80 tx=0x7f1560009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1560038070 con 0x7f15700fc2b0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1560032970 con 0x7f15700fc2b0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1570070110 con 0x7f15700fc2b0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.377+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1560033860 con 0x7f15700fc2b0 2024-06-13T01:23:56.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.378+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1570114f00 con 0x7f15700fc2b0 2024-06-13T01:23:56.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.379+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15700fb710 con 0x7f15700fc2b0 2024-06-13T01:23:56.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.379+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1560030020 con 0x7f15700fc2b0 2024-06-13T01:23:56.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.379+0000 7f1566ffd640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f155c065430 0x7f155c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:56.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.379+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f15600a2340 con 0x7f15700fc2b0 2024-06-13T01:23:56.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.380+0000 7f1567fff640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f155c065430 0x7f155c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:56.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.380+0000 7f1567fff640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f155c065430 0x7f155c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1558004ed0 tx=0x7f155800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:56.382 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.384+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f15600708a0 con 0x7f15700fc2b0 2024-06-13T01:23:56.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.614+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1570104e20 con 0x7f15700fc2b0 2024-06-13T01:23:56.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.617+0000 7f1566ffd640 1 -- 172.21.15.132:0/1092596419 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f15600410f0 con 0x7f15700fc2b0 2024-06-13T01:23:56.617 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-06-13T01:23:56.617 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQAsSmpmiyqvJBAAR8rIRa3BVDKcPkMODuxbJA== 2024-06-13T01:23:56.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f155c065430 msgr2=0x7f155c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:56.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f155c065430 0x7f155c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1558004ed0 tx=0x7f155800b040 comp rx=0 tx=0).stop 2024-06-13T01:23:56.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 msgr2=0x7f15700712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700712d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1560009f80 tx=0x7f1560009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 shutdown_connections 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f155c065430 0x7f155c0678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15700fd580 0x7f157006f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 --2- 172.21.15.132:0/1092596419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15700fc2b0 0x7f15700712d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 >> 172.21.15.132:0/1092596419 conn(0x7f15700f7a70 msgr2=0x7f15700f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 shutdown_connections 2024-06-13T01:23:56.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:23:56.620+0000 7f1574a90640 1 -- 172.21.15.132:0/1092596419 wait complete. 2024-06-13T01:23:56.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:56 smithi132 ceph-mon[29330]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:56.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:56 smithi155 ceph-mon[35815]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:57.363 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-06-13T01:23:57.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-13T01:23:57.363 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-13T01:23:57.401 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-13T01:23:57.664 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi155/config 2024-06-13T01:23:57.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:57 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1092596419' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T01:23:57.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:57 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1092596419' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T01:23:57.787 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:57 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1092596419' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T01:23:57.787 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:57 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1092596419' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T01:23:58.710 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:58 smithi132 ceph-mon[29330]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:58.753 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:58 smithi155 ceph-mon[35815]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:23:59.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 -- 172.21.15.155:0/1943690377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 msgr2=0x7f82a80fa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:59.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 --2- 172.21.15.155:0/1943690377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a80fa3d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f82900048d0 tx=0x7f8290009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 -- 172.21.15.155:0/1943690377 shutdown_connections 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 --2- 172.21.15.155:0/1943690377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a80fa3d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 --2- 172.21.15.155:0/1943690377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82a80fd920 0x7f82a80fdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 -- 172.21.15.155:0/1943690377 >> 172.21.15.155:0/1943690377 conn(0x7f82a80f6100 msgr2=0x7f82a80f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 -- 172.21.15.155:0/1943690377 shutdown_connections 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.285+0000 7f82ae144640 1 -- 172.21.15.155:0/1943690377 wait complete. 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 Processor -- start 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 -- start start 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82a80fd920 0x7f82a810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a810d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a810da50 con 0x7f82a80fd920 2024-06-13T01:23:59.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a810e270 con 0x7f82a80fe270 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a810d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a810d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:52518/0 (socket says 172.21.15.155:52518) 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 -- 172.21.15.155:0/1316264911 learned_addr learned my addr 172.21.15.155:0/1316264911 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 -- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82a80fd920 msgr2=0x7f82a810cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82a77fe640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82a80fd920 0x7f82a810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82a80fd920 0x7f82a810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 -- 172.21.15.155:0/1316264911 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8290005040 con 0x7f82a80fe270 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f829ffff640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a810d400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8290000c00 tx=0x7f8290009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8290038070 con 0x7f82a80fe270 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.286+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a810e4a0 con 0x7f82a80fe270 2024-06-13T01:23:59.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.287+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8290032970 con 0x7f82a80fe270 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.287+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8290033950 con 0x7f82a80fe270 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.287+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a810e9e0 con 0x7f82a80fe270 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.287+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a80f9dc0 con 0x7f82a80fe270 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.288+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8290030020 con 0x7f82a80fe270 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.288+0000 7f82a4ff9640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8294065430 0x7f82940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.288+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f82900a2550 con 0x7f82a80fe270 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.289+0000 7f82a77fe640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8294065430 0x7f82940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.289+0000 7f82a77fe640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8294065430 0x7f82940678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8298007fd0 tx=0x7f8298009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:23:59.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.292+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8290070a00 con 0x7f82a80fe270 2024-06-13T01:23:59.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.522+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f82a804b480 con 0x7f82a80fe270 2024-06-13T01:23:59.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.527+0000 7f82a4ff9640 1 -- 172.21.15.155:0/1316264911 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8290074020 con 0x7f82a80fe270 2024-06-13T01:23:59.526 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-06-13T01:23:59.526 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAvSmpmjUc8HxAAccwgqrprFBaG7fOLLB5AIA== 2024-06-13T01:23:59.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8294065430 msgr2=0x7f82940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:59.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8294065430 0x7f82940678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8298007fd0 tx=0x7f8298009040 comp rx=0 tx=0).stop 2024-06-13T01:23:59.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 msgr2=0x7f82a810d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:23:59.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a810d400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8290000c00 tx=0x7f8290009da0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 shutdown_connections 2024-06-13T01:23:59.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8294065430 0x7f82940678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82a80fe270 0x7f82a810d400 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 --2- 172.21.15.155:0/1316264911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82a80fd920 0x7f82a810cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:23:59.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 >> 172.21.15.155:0/1316264911 conn(0x7f82a80f6100 msgr2=0x7f82a80f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:23:59.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 shutdown_connections 2024-06-13T01:23:59.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-06-13T01:23:59.529+0000 7f82ae144640 1 -- 172.21.15.155:0/1316264911 wait complete. 2024-06-13T01:23:59.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:59 smithi155 ceph-mon[35815]: from='client.? 172.21.15.155:0/1316264911' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T01:23:59.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:59 smithi155 ceph-mon[35815]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T01:23:59.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:23:59 smithi155 ceph-mon[35815]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T01:23:59.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:59 smithi132 ceph-mon[29330]: from='client.? 172.21.15.155:0/1316264911' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T01:23:59.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:59 smithi132 ceph-mon[29330]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-13T01:23:59.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:23:59 smithi132 ceph-mon[29330]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-13T01:24:00.243 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-06-13T01:24:00.243 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-13T01:24:00.243 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-13T01:24:00.280 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-13T01:24:00.280 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-13T01:24:00.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph mgr dump --format=json 2024-06-13T01:24:00.542 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:00 smithi155 ceph-mon[35815]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:00.947 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:00 smithi132 ceph-mon[29330]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:02.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 -- 172.21.15.132:0/2548573425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 msgr2=0x7fc7540fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 --2- 172.21.15.132:0/2548573425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc7540fb940 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc7440048d0 tx=0x7fc744009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 -- 172.21.15.132:0/2548573425 shutdown_connections 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 --2- 172.21.15.132:0/2548573425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 0x7fc7540fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 --2- 172.21.15.132:0/2548573425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc7540fb940 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 -- 172.21.15.132:0/2548573425 >> 172.21.15.132:0/2548573425 conn(0x7fc754072110 msgr2=0x7fc754072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 -- 172.21.15.132:0/2548573425 shutdown_connections 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.217+0000 7fc758983640 1 -- 172.21.15.132:0/2548573425 wait complete. 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc758983640 1 Processor -- start 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc758983640 1 -- start start 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc758983640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc754111f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc758983640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 0x7fc7541124b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:02.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc758983640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754112b40 con 0x7fc7540fb570 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc758983640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75410bf40 con 0x7fc7540fbe80 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc754111f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc754111f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43766/0 (socket says 172.21.15.132:43766) 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 -- 172.21.15.132:0/3485722039 learned_addr learned my addr 172.21.15.132:0/3485722039 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc749d9b640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 0x7fc7541124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 -- 172.21.15.132:0/3485722039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 msgr2=0x7fc7541124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 0x7fc7541124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 -- 172.21.15.132:0/3485722039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc744005040 con 0x7fc7540fb570 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc749d9b640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 0x7fc7541124b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.218+0000 7fc75259c640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc754111f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc744003080 tx=0x7fc744009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.219+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc744038070 con 0x7fc7540fb570 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.219+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc744008ee0 con 0x7fc7540fb570 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.219+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75410c1c0 con 0x7fc7540fb570 2024-06-13T01:24:02.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.219+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc74403d870 con 0x7fc7540fb570 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.219+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75410c680 con 0x7fc7540fb570 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.220+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7540fc920 con 0x7fc7540fb570 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.220+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc744030020 con 0x7fc7540fb570 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.220+0000 7fc74b7fe640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc740065430 0x7fc7400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.220+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc7440a2160 con 0x7fc7540fb570 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.220+0000 7fc749d9b640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc740065430 0x7fc7400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.221+0000 7fc749d9b640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc740065430 0x7fc7400678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc75410d830 tx=0x7fc73c008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:02.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.225+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc7440707a0 con 0x7fc7540fb570 2024-06-13T01:24:02.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.435+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc75404b4e0 con 0x7fc7540fb570 2024-06-13T01:24:02.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.437+0000 7fc74b7fe640 1 -- 172.21.15.132:0/3485722039 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161456 (secure 0 0 0) 0x7fc744073dc0 con 0x7fc7540fb570 2024-06-13T01:24:02.439 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:02.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc740065430 msgr2=0x7fc7400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:02.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc740065430 0x7fc7400678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc75410d830 tx=0x7fc73c008040 comp rx=0 tx=0).stop 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 msgr2=0x7fc754111f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc754111f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc744003080 tx=0x7fc744009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 shutdown_connections 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc740065430 0x7fc7400678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540fbe80 0x7fc7541124b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 --2- 172.21.15.132:0/3485722039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540fb570 0x7fc754111f70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 >> 172.21.15.132:0/3485722039 conn(0x7fc754072110 msgr2=0x7fc7540f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 shutdown_connections 2024-06-13T01:24:02.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:02.440+0000 7fc758983640 1 -- 172.21.15.132:0/3485722039 wait complete. 2024-06-13T01:24:02.810 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:02 smithi132 ceph-mon[29330]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:02.810 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:02 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3485722039' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-13T01:24:03.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:02 smithi155 ceph-mon[35815]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:03.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:02 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3485722039' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-13T01:24:03.193 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":19,"flags":0,"active_gid":14210,"active_name":"smithi132.vjweoa","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2992514013},{"type":"v1","addr":"172.21.15.132:6801","nonce":2992514013}]},"active_addr":"172.21.15.132:6801/2992514013","active_change":"2024-06-13T01:20:19.303510+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi155.teuqcw","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.132:8443/","prometheus":"http://172.21.15.132:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":3671091774}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":1513999218}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":3085347119}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":2650084977}]}]} 2024-06-13T01:24:03.196 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-13T01:24:03.196 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-13T01:24:03.197 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T01:24:03.455 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:04.943 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:04 smithi132 ceph-mon[29330]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:04.943 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:05.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:04 smithi155 ceph-mon[35815]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:05.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:05.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 -- 172.21.15.132:0/923755975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 msgr2=0x7f38a00fb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 --2- 172.21.15.132:0/923755975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a00fb1d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f38940048d0 tx=0x7f3894009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 -- 172.21.15.132:0/923755975 shutdown_connections 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 --2- 172.21.15.132:0/923755975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a00fb1d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 --2- 172.21.15.132:0/923755975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 0x7f38a0101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 -- 172.21.15.132:0/923755975 >> 172.21.15.132:0/923755975 conn(0x7f38a00f6d90 msgr2=0x7f38a00f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 -- 172.21.15.132:0/923755975 shutdown_connections 2024-06-13T01:24:05.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.159+0000 7f38a54f3640 1 -- 172.21.15.132:0/923755975 wait complete. 2024-06-13T01:24:05.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f38a54f3640 1 Processor -- start 2024-06-13T01:24:05.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f38a54f3640 1 -- start start 2024-06-13T01:24:05.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f38a54f3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 0x7f38a010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f38a54f3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a010e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f38a54f3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a010ec40 con 0x7f38a0101b40 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f38a54f3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a010edb0 con 0x7f38a01011a0 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389e7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a010e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389e7fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a010e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43786/0 (socket says 172.21.15.132:43786) 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389e7fc640 1 -- 172.21.15.132:0/3405544277 learned_addr learned my addr 172.21.15.132:0/3405544277 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389effd640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 0x7f38a010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:05.162 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389e7fc640 1 -- 172.21.15.132:0/3405544277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 msgr2=0x7f38a010e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389e7fc640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 0x7f38a010e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389e7fc640 1 -- 172.21.15.132:0/3405544277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894005040 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.160+0000 7f389effd640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 0x7f38a010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.161+0000 7f389e7fc640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a010e560 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3894003080 tx=0x7f3894009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.161+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3894038070 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.161+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3894008ee0 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.161+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a0117090 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.161+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f389403d870 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.161+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a0117500 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.162+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a00fc910 con 0x7f38a0101b40 2024-06-13T01:24:05.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.162+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3894030020 con 0x7f38a0101b40 2024-06-13T01:24:05.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.163+0000 7f38877fe640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3888065430 0x7f38880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:05.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.163+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f38940a2190 con 0x7f38a0101b40 2024-06-13T01:24:05.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.163+0000 7f389effd640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3888065430 0x7f38880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:05.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.163+0000 7f389effd640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3888065430 0x7f38880678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3890007930 tx=0x7f3890008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:05.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.167+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3894070760 con 0x7f38a0101b40 2024-06-13T01:24:05.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.332+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38a004b480 con 0x7f38a0101b40 2024-06-13T01:24:05.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.332+0000 7f38877fe640 1 -- 172.21.15.132:0/3405544277 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11123 (secure 0 0 0) 0x7f3894073d80 con 0x7f38a0101b40 2024-06-13T01:24:05.332 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:05.332 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":40,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","created":"2024-06-13T01:18:26.791525+0000","modified":"2024-06-13T01:23:45.740129+0000","last_up_change":"2024-06-13T01:23:44.734688+0000","last_in_change":"2024-06-13T01:23:25.927571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T01:22:29.617177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c3479cf5-50de-49b8-8219-8373f5329897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6803","nonce":3154009542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6805","nonce":3154009542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6809","nonce":3154009542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6807","nonce":3154009542}]},"public_addr":"172.21.15.132:6803/3154009542","cluster_addr":"172.21.15.132:6805/3154009542","heartbeat_back_addr":"172.21.15.132:6809/3154009542","heartbeat_front_addr":"172.21.15.132:6807/3154009542","state":["exists","up"]},{"osd":1,"uuid":"77f55f5e-4836-40db-9b9c-712be934facf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6811","nonce":2195816014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6813","nonce":2195816014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6817","nonce":2195816014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6815","nonce":2195816014}]},"public_addr":"172.21.15.132:6811/2195816014","cluster_addr":"172.21.15.132:6813/2195816014","heartbeat_back_addr":"172.21.15.132:6817/2195816014","heartbeat_front_addr":"172.21.15.132:6815/2195816014","state":["exists","up"]},{"osd":2,"uuid":"6fe9868e-7649-4077-8868-d77818e0aedc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6819","nonce":2883035986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6821","nonce":2883035986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6825","nonce":2883035986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6823","nonce":2883035986}]},"public_addr":"172.21.15.132:6819/2883035986","cluster_addr":"172.21.15.132:6821/2883035986","heartbeat_back_addr":"172.21.15.132:6825/2883035986","heartbeat_front_addr":"172.21.15.132:6823/2883035986","state":["exists","up"]},{"osd":3,"uuid":"747fee8a-d594-4e25-afb9-fd1c0fe56083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6801","nonce":920787201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6803","nonce":920787201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6807","nonce":920787201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6805","nonce":920787201}]},"public_addr":"172.21.15.155:6801/920787201","cluster_addr":"172.21.15.155:6803/920787201","heartbeat_back_addr":"172.21.15.155:6807/920787201","heartbeat_front_addr":"172.21.15.155:6805/920787201","state":["exists","up"]},{"osd":4,"uuid":"aa586deb-7be2-4b28-a48c-8bb58513e492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6809","nonce":3401072403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6811","nonce":3401072403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6815","nonce":3401072403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6813","nonce":3401072403}]},"public_addr":"172.21.15.155:6809/3401072403","cluster_addr":"172.21.15.155:6811/3401072403","heartbeat_back_addr":"172.21.15.155:6815/3401072403","heartbeat_front_addr":"172.21.15.155:6813/3401072403","state":["exists","up"]},{"osd":5,"uuid":"b26458ee-737c-4e2b-a627-793ba7098bbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6817","nonce":410629247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6819","nonce":410629247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6823","nonce":410629247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6821","nonce":410629247}]},"public_addr":"172.21.15.155:6817/410629247","cluster_addr":"172.21.15.155:6819/410629247","heartbeat_back_addr":"172.21.15.155:6823/410629247","heartbeat_front_addr":"172.21.15.155:6821/410629247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:21:32.386358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:00.506581+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:27.579430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:51.894977+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:23:17.195060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:23:43.496687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.132:6800/3829496564":"2024-06-14T01:20:19.303360+0000","172.21.15.132:6801/3829496564":"2024-06-14T01:20:19.303360+0000","172.21.15.132:0/1381082418":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6800/1802099589":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6801/1802099589":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6801/2668291289":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/2054197342":"2024-06-14T01:20:19.303360+0000","172.21.15.132:0/2808306670":"2024-06-14T01:18:48.373793+0000","172.21.15.132:0/331608642":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/2828217670":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/3751223226":"2024-06-14T01:18:48.373793+0000","172.21.15.132:0/4280057465":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/1942975292":"2024-06-14T01:20:19.303360+0000","172.21.15.132:6800/2668291289":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/3189205445":"2024-06-14T01:20:19.303360+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.334+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3888065430 msgr2=0x7f38880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.334+0000 7f38a54f3640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3888065430 0x7f38880678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3890007930 tx=0x7f3890008040 comp rx=0 tx=0).stop 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 msgr2=0x7f38a010e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a010e560 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3894003080 tx=0x7f3894009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 shutdown_connections 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3888065430 0x7f38880678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38a0101b40 0x7f38a010e560 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 --2- 172.21.15.132:0/3405544277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38a01011a0 0x7f38a010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:05.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 >> 172.21.15.132:0/3405544277 conn(0x7f38a00f6d90 msgr2=0x7f38a00f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:05.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 shutdown_connections 2024-06-13T01:24:05.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:05.335+0000 7f38a54f3640 1 -- 172.21.15.132:0/3405544277 wait complete. 2024-06-13T01:24:05.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:05 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3405544277' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T01:24:05.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:05 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3405544277' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T01:24:06.124 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-13T01:24:06.125 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd dump --format=json 2024-06-13T01:24:06.376 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:06.821 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:06 smithi132 ceph-mon[29330]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:07.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:06 smithi155 ceph-mon[35815]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 -- 172.21.15.132:0/3449995065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af01022c0 msgr2=0x7f7af0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 --2- 172.21.15.132:0/3449995065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af01022c0 0x7f7af0102690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0048d0 tx=0x7f7aec009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 -- 172.21.15.132:0/3449995065 shutdown_connections 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 --2- 172.21.15.132:0/3449995065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7af0102c60 0x7f7af00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 --2- 172.21.15.132:0/3449995065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af01022c0 0x7f7af0102690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 -- 172.21.15.132:0/3449995065 >> 172.21.15.132:0/3449995065 conn(0x7f7af00f7e50 msgr2=0x7f7af00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:07.981 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 -- 172.21.15.132:0/3449995065 shutdown_connections 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.980+0000 7f7af7649640 1 -- 172.21.15.132:0/3449995065 wait complete. 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af7649640 1 Processor -- start 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af7649640 1 -- start start 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af7649640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7af01022c0 0x7f7af01133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af7649640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 0x7f7af0113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af7649640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af0113fa0 con 0x7f7af0102c60 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af7649640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af010d350 con 0x7f7af01022c0 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af4be4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 0x7f7af0113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af4be4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 0x7f7af0113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43790/0 (socket says 172.21.15.132:43790) 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af4be4640 1 -- 172.21.15.132:0/3421662441 learned_addr learned my addr 172.21.15.132:0/3421662441 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af53e5640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7af01022c0 0x7f7af01133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:07.982 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af4be4640 1 -- 172.21.15.132:0/3421662441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7af01022c0 msgr2=0x7f7af01133d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af4be4640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7af01022c0 0x7f7af01133d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.981+0000 7f7af4be4640 1 -- 172.21.15.132:0/3421662441 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec005040 con 0x7f7af0102c60 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.982+0000 7f7af4be4640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 0x7f7af0113910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000b790 tx=0x7f7ae00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.982+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ae000c700 con 0x7f7af0102c60 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.982+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ae000cd30 con 0x7f7af0102c60 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.982+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ae0013370 con 0x7f7af0102c60 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.982+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af010d630 con 0x7f7af0102c60 2024-06-13T01:24:07.983 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.982+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af010db00 con 0x7f7af0102c60 2024-06-13T01:24:07.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.983+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af00fda30 con 0x7f7af0102c60 2024-06-13T01:24:07.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.983+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7ae0012070 con 0x7f7af0102c60 2024-06-13T01:24:07.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.984+0000 7f7ae5ffb640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7adc065430 0x7f7adc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:07.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.984+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7ae0081dd0 con 0x7f7af0102c60 2024-06-13T01:24:07.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.984+0000 7f7af53e5640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7adc065430 0x7f7adc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:07.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.984+0000 7f7af53e5640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7adc065430 0x7f7adc0678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009f80 tx=0x7f7aec035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:07.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:07.988+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7ae0050280 con 0x7f7af0102c60 2024-06-13T01:24:08.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.152+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7af00fbab0 con 0x7f7af0102c60 2024-06-13T01:24:08.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.153+0000 7f7ae5ffb640 1 -- 172.21.15.132:0/3421662441 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11123 (secure 0 0 0) 0x7f7ae00538a0 con 0x7f7af0102c60 2024-06-13T01:24:08.152 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:08.152 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":40,"fsid":"ad8d4854-2922-11ef-bc9c-c7b262605968","created":"2024-06-13T01:18:26.791525+0000","modified":"2024-06-13T01:23:45.740129+0000","last_up_change":"2024-06-13T01:23:44.734688+0000","last_in_change":"2024-06-13T01:23:25.927571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-13T01:22:29.617177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c3479cf5-50de-49b8-8219-8373f5329897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6803","nonce":3154009542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6805","nonce":3154009542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6809","nonce":3154009542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3154009542},{"type":"v1","addr":"172.21.15.132:6807","nonce":3154009542}]},"public_addr":"172.21.15.132:6803/3154009542","cluster_addr":"172.21.15.132:6805/3154009542","heartbeat_back_addr":"172.21.15.132:6809/3154009542","heartbeat_front_addr":"172.21.15.132:6807/3154009542","state":["exists","up"]},{"osd":1,"uuid":"77f55f5e-4836-40db-9b9c-712be934facf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6811","nonce":2195816014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6813","nonce":2195816014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6817","nonce":2195816014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":2195816014},{"type":"v1","addr":"172.21.15.132:6815","nonce":2195816014}]},"public_addr":"172.21.15.132:6811/2195816014","cluster_addr":"172.21.15.132:6813/2195816014","heartbeat_back_addr":"172.21.15.132:6817/2195816014","heartbeat_front_addr":"172.21.15.132:6815/2195816014","state":["exists","up"]},{"osd":2,"uuid":"6fe9868e-7649-4077-8868-d77818e0aedc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6819","nonce":2883035986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6821","nonce":2883035986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6825","nonce":2883035986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2883035986},{"type":"v1","addr":"172.21.15.132:6823","nonce":2883035986}]},"public_addr":"172.21.15.132:6819/2883035986","cluster_addr":"172.21.15.132:6821/2883035986","heartbeat_back_addr":"172.21.15.132:6825/2883035986","heartbeat_front_addr":"172.21.15.132:6823/2883035986","state":["exists","up"]},{"osd":3,"uuid":"747fee8a-d594-4e25-afb9-fd1c0fe56083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6801","nonce":920787201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6803","nonce":920787201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6807","nonce":920787201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":920787201},{"type":"v1","addr":"172.21.15.155:6805","nonce":920787201}]},"public_addr":"172.21.15.155:6801/920787201","cluster_addr":"172.21.15.155:6803/920787201","heartbeat_back_addr":"172.21.15.155:6807/920787201","heartbeat_front_addr":"172.21.15.155:6805/920787201","state":["exists","up"]},{"osd":4,"uuid":"aa586deb-7be2-4b28-a48c-8bb58513e492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6809","nonce":3401072403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6811","nonce":3401072403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6815","nonce":3401072403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3401072403},{"type":"v1","addr":"172.21.15.155:6813","nonce":3401072403}]},"public_addr":"172.21.15.155:6809/3401072403","cluster_addr":"172.21.15.155:6811/3401072403","heartbeat_back_addr":"172.21.15.155:6815/3401072403","heartbeat_front_addr":"172.21.15.155:6813/3401072403","state":["exists","up"]},{"osd":5,"uuid":"b26458ee-737c-4e2b-a627-793ba7098bbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6817","nonce":410629247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6819","nonce":410629247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6823","nonce":410629247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":410629247},{"type":"v1","addr":"172.21.15.155:6821","nonce":410629247}]},"public_addr":"172.21.15.155:6817/410629247","cluster_addr":"172.21.15.155:6819/410629247","heartbeat_back_addr":"172.21.15.155:6823/410629247","heartbeat_front_addr":"172.21.15.155:6821/410629247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:21:32.386358+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:00.506581+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:27.579430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:22:51.894977+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:23:17.195060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-13T01:23:43.496687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.132:6800/3829496564":"2024-06-14T01:20:19.303360+0000","172.21.15.132:6801/3829496564":"2024-06-14T01:20:19.303360+0000","172.21.15.132:0/1381082418":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6800/1802099589":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6801/1802099589":"2024-06-14T01:18:48.373793+0000","172.21.15.132:6801/2668291289":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/2054197342":"2024-06-14T01:20:19.303360+0000","172.21.15.132:0/2808306670":"2024-06-14T01:18:48.373793+0000","172.21.15.132:0/331608642":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/2828217670":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/3751223226":"2024-06-14T01:18:48.373793+0000","172.21.15.132:0/4280057465":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/1942975292":"2024-06-14T01:20:19.303360+0000","172.21.15.132:6800/2668291289":"2024-06-14T01:19:17.499154+0000","172.21.15.132:0/3189205445":"2024-06-14T01:20:19.303360+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-13T01:24:08.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7adc065430 msgr2=0x7f7adc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:08.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7adc065430 0x7f7adc0678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009f80 tx=0x7f7aec035040 comp rx=0 tx=0).stop 2024-06-13T01:24:08.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 msgr2=0x7f7af0113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:08.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 0x7f7af0113910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000b790 tx=0x7f7ae00091d0 comp rx=0 tx=0).stop 2024-06-13T01:24:08.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 shutdown_connections 2024-06-13T01:24:08.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7adc065430 0x7f7adc0678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:08.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7af0102c60 0x7f7af0113910 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:08.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 --2- 172.21.15.132:0/3421662441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7af01022c0 0x7f7af01133d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:08.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 >> 172.21.15.132:0/3421662441 conn(0x7f7af00f7e50 msgr2=0x7f7af00f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:08.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 shutdown_connections 2024-06-13T01:24:08.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:08.155+0000 7f7af7649640 1 -- 172.21.15.132:0/3421662441 wait complete. 2024-06-13T01:24:08.757 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:08 smithi132 ceph-mon[29330]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:08.758 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:08 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3421662441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T01:24:08.782 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-13T01:24:08.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-13T01:24:08.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-13T01:24:08.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-13T01:24:08.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-13T01:24:08.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-13T01:24:09.092 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:09.104 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:09.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:08 smithi155 ceph-mon[35815]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:09.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:08 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3421662441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-13T01:24:10.444 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:10.903 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:10 smithi132 ceph-mon[29330]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:10.935 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:10 smithi155 ceph-mon[35815]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:11.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 -- 172.21.15.132:0/300900989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e241022a0 msgr2=0x7f7e24102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/300900989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e241022a0 0x7f7e24102670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e200048d0 tx=0x7f7e20009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 -- 172.21.15.132:0/300900989 shutdown_connections 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/300900989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e24102c40 0x7f7e240fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/300900989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e241022a0 0x7f7e24102670 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 -- 172.21.15.132:0/300900989 >> 172.21.15.132:0/300900989 conn(0x7f7e240f7e70 msgr2=0x7f7e240fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 -- 172.21.15.132:0/300900989 shutdown_connections 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.361+0000 7f7e2b40e640 1 -- 172.21.15.132:0/300900989 wait complete. 2024-06-13T01:24:11.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e2b40e640 1 Processor -- start 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e2b40e640 1 -- start start 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e2b40e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e241022a0 0x7f7e24113090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e2b40e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 0x7f7e241135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e2b40e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24113c60 con 0x7f7e24102c40 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e2b40e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2410d010 con 0x7f7e241022a0 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e289a9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 0x7f7e241135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e289a9640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 0x7f7e241135d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43798/0 (socket says 172.21.15.132:43798) 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e289a9640 1 -- 172.21.15.132:0/961033501 learned_addr learned my addr 172.21.15.132:0/961033501 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:11.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.362+0000 7f7e291aa640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e241022a0 0x7f7e24113090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e289a9640 1 -- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e241022a0 msgr2=0x7f7e24113090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e289a9640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e241022a0 0x7f7e24113090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e289a9640 1 -- 172.21.15.132:0/961033501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e20005040 con 0x7f7e24102c40 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e289a9640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 0x7f7e241135d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400b790 tx=0x7f7e140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e1400c700 con 0x7f7e24102c40 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e1400cd30 con 0x7f7e24102c40 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2410d2f0 con 0x7f7e24102c40 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e14013370 con 0x7f7e24102c40 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.363+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2410d810 con 0x7f7e24102c40 2024-06-13T01:24:11.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.364+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e2410dd50 con 0x7f7e24102c40 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.364+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7e14012070 con 0x7f7e24102c40 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e19ffb640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7e10065430 0x7f7e100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e14081dd0 con 0x7f7e24102c40 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e291aa640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7e10065430 0x7f7e100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e19ffb640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] conn(0x7f7e1006b1b0 0x7f7e1006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 --> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e1006dc70 con 0x7f7e1006b1b0 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e14087430 con 0x7f7e24102c40 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e291aa640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7e10065430 0x7f7e100678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009f80 tx=0x7f7e20035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.365+0000 7f7e299ab640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] conn(0x7f7e1006b1b0 0x7f7e1006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:11.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.366+0000 7f7e299ab640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] conn(0x7f7e1006b1b0 0x7f7e1006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:11.370 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.367+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== osd.3 v2:172.21.15.155:6800/920787201 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7e1006dc70 con 0x7f7e1006b1b0 2024-06-13T01:24:11.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.380+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 --> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e240f9780 con 0x7f7e1006b1b0 2024-06-13T01:24:11.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.380+0000 7f7e19ffb640 1 -- 172.21.15.132:0/961033501 <== osd.3 v2:172.21.15.155:6800/920787201 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7e240f9780 con 0x7f7e1006b1b0 2024-06-13T01:24:11.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] conn(0x7f7e1006b1b0 msgr2=0x7f7e1006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] conn(0x7f7e1006b1b0 0x7f7e1006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7e10065430 msgr2=0x7f7e100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7e10065430 0x7f7e100678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009f80 tx=0x7f7e20035040 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 msgr2=0x7f7e241135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 0x7f7e241135d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400b790 tx=0x7f7e140091d0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 shutdown_connections 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7e10065430 0x7f7e100678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e24102c40 0x7f7e241135d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:6800/920787201,v1:172.21.15.155:6801/920787201] conn(0x7f7e1006b1b0 0x7f7e1006d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 --2- 172.21.15.132:0/961033501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e241022a0 0x7f7e24113090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 >> 172.21.15.132:0/961033501 conn(0x7f7e240f7e70 msgr2=0x7f7e240f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 shutdown_connections 2024-06-13T01:24:11.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:11.381+0000 7f7e2b40e640 1 -- 172.21.15.132:0/961033501 wait complete. 2024-06-13T01:24:12.035 INFO:teuthology.orchestra.run.smithi132.stdout:111669149714 2024-06-13T01:24:12.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-13T01:24:12.843 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:12.843 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:12.852 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:12 smithi132 ceph-mon[29330]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:13.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:12 smithi155 ceph-mon[35815]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:13.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 -- 172.21.15.132:0/3283083266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f006f810 msgr2=0x7fc8f006fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/3283083266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f006f810 0x7fc8f006fbe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0048d0 tx=0x7fc8ec009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 -- 172.21.15.132:0/3283083266 shutdown_connections 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/3283083266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f0070120 0x7fc8f01085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/3283083266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f006f810 0x7fc8f006fbe0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 -- 172.21.15.132:0/3283083266 >> 172.21.15.132:0/3283083266 conn(0x7fc8f00f7e50 msgr2=0x7fc8f00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 -- 172.21.15.132:0/3283083266 shutdown_connections 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.253+0000 7fc8f6c10640 1 -- 172.21.15.132:0/3283083266 wait complete. 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8f6c10640 1 Processor -- start 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8f6c10640 1 -- start start 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8f6c10640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f006f810 0x7fc8f0115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:13.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8f6c10640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 0x7fc8f01157a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8f6c10640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0115da0 con 0x7fc8f0070120 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8f6c10640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f010f1e0 con 0x7fc8f006f810 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8e7fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 0x7fc8f01157a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8e7fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 0x7fc8f01157a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59314/0 (socket says 172.21.15.132:59314) 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.254+0000 7fc8e7fff640 1 -- 172.21.15.132:0/1572464863 learned_addr learned my addr 172.21.15.132:0/1572464863 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8f49ac640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f006f810 0x7fc8f0115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e7fff640 1 -- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f006f810 msgr2=0x7fc8f0115260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e7fff640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f006f810 0x7fc8f0115260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e7fff640 1 -- 172.21.15.132:0/1572464863 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec005040 con 0x7fc8f0070120 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8f49ac640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f006f810 0x7fc8f0115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:24:13.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e7fff640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 0x7fc8f01157a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0002a80 tx=0x7fc8e0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e0007c30 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e000f040 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f010f4c0 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e00096b0 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.255+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f010f9e0 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.256+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc8f010fec0 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.256+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc8e000d6b0 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8e57fa640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc8d8065430 0x7fc8d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc8e0081f60 con 0x7fc8f0070120 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8f49ac640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc8d8065430 0x7fc8d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8e57fa640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] conn(0x7fc8d806b1b0 0x7fc8d806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:13.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 --> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8d806dc70 con 0x7fc8d806b1b0 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8e0084520 con 0x7fc8f0070120 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8f49ac640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc8d8065430 0x7fc8d80678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec000c00 tx=0x7fc8ec009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.257+0000 7fc8f51ad640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] conn(0x7fc8d806b1b0 0x7fc8d806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.258+0000 7fc8f51ad640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] conn(0x7fc8d806b1b0 0x7fc8d806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.259+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== osd.4 v2:172.21.15.155:6808/3401072403 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc8d806dc70 con 0x7fc8d806b1b0 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 --> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc8f00f9c80 con 0x7fc8d806b1b0 2024-06-13T01:24:13.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8e57fa640 1 -- 172.21.15.132:0/1572464863 <== osd.4 v2:172.21.15.155:6808/3401072403 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc8f00f9c80 con 0x7fc8d806b1b0 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] conn(0x7fc8d806b1b0 msgr2=0x7fc8d806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] conn(0x7fc8d806b1b0 0x7fc8d806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc8d8065430 msgr2=0x7fc8d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc8d8065430 0x7fc8d80678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec000c00 tx=0x7fc8ec009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 msgr2=0x7fc8f01157a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.272+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 0x7fc8f01157a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0002a80 tx=0x7fc8e0004600 comp rx=0 tx=0).stop 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 shutdown_connections 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc8d8065430 0x7fc8d80678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8f0070120 0x7fc8f01157a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:6808/3401072403,v1:172.21.15.155:6809/3401072403] conn(0x7fc8d806b1b0 0x7fc8d806d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 --2- 172.21.15.132:0/1572464863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8f006f810 0x7fc8f0115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:13.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 >> 172.21.15.132:0/1572464863 conn(0x7fc8f00f7e50 msgr2=0x7fc8f00f92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:13.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 shutdown_connections 2024-06-13T01:24:13.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:13.273+0000 7fc8f6c10640 1 -- 172.21.15.132:0/1572464863 wait complete. 2024-06-13T01:24:13.470 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:13.889 INFO:teuthology.orchestra.run.smithi132.stdout:146028888077 2024-06-13T01:24:13.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-13T01:24:14.920 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:14 smithi132 ceph-mon[29330]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:15.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:14 smithi155 ceph-mon[35815]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:15.329 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:16.696 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:16 smithi132 ceph-mon[29330]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:17.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:16 smithi155 ceph-mon[35815]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:18.570 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:18.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:18 smithi132 ceph-mon[29330]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 -- 172.21.15.132:0/682586488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 msgr2=0x7f33d00fb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 --2- 172.21.15.132:0/682586488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d00fb1d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f33c00048d0 tx=0x7f33c0009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 -- 172.21.15.132:0/682586488 shutdown_connections 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 --2- 172.21.15.132:0/682586488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d00fb1d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 --2- 172.21.15.132:0/682586488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 0x7f33d0101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 -- 172.21.15.132:0/682586488 >> 172.21.15.132:0/682586488 conn(0x7f33d00f6d90 msgr2=0x7f33d00f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 -- 172.21.15.132:0/682586488 shutdown_connections 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 -- 172.21.15.132:0/682586488 wait complete. 2024-06-13T01:24:19.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:18.999+0000 7f33d7d19640 1 Processor -- start 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d7d19640 1 -- start start 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d7d19640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 0x7f33d010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d7d19640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d7d19640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d010eb40 con 0x7f33d0101b40 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d7d19640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d010ecb0 con 0x7f33d01011a0 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59326/0 (socket says 172.21.15.132:59326) 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 -- 172.21.15.132:0/1936985005 learned_addr learned my addr 172.21.15.132:0/1936985005 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d5ab5640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 0x7f33d010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 -- 172.21.15.132:0/1936985005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 msgr2=0x7f33d010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:19.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 0x7f33d010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 -- 172.21.15.132:0/1936985005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0005040 con 0x7f33d0101b40 2024-06-13T01:24:19.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d5ab5640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 0x7f33d010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:24:19.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.000+0000 7f33d52b4640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d010e4f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f33c00043d0 tx=0x7f33c0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:19.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.001+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33c0038070 con 0x7f33d0101b40 2024-06-13T01:24:19.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.001+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33c0032970 con 0x7f33d0101b40 2024-06-13T01:24:19.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.001+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d0117020 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.001+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33c003da60 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.001+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d0117480 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.002+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33d00f8770 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f33c0030020 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33c67fc640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f33bc065430 0x7f33bc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f33c00a2590 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33d5ab5640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f33bc065430 0x7f33bc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33c67fc640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] conn(0x7f33bc06b1b0 0x7f33bc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 --> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33bc06dc70 con 0x7f33bc06b1b0 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.003+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33c00a4be0 con 0x7f33d0101b40 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.004+0000 7f33d62b6640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] conn(0x7f33bc06b1b0 0x7f33bc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:19.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.004+0000 7f33d5ab5640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f33bc065430 0x7f33bc0678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f33cc004d80 tx=0x7f33cc00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.004+0000 7f33d62b6640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] conn(0x7f33bc06b1b0 0x7f33bc06d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.005+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== osd.1 v2:172.21.15.132:6810/2195816014 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f33bc06dc70 con 0x7f33bc06b1b0 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 --> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33d00fc910 con 0x7f33bc06b1b0 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33c67fc640 1 -- 172.21.15.132:0/1936985005 <== osd.1 v2:172.21.15.132:6810/2195816014 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33d00fc910 con 0x7f33bc06b1b0 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] conn(0x7f33bc06b1b0 msgr2=0x7f33bc06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] conn(0x7f33bc06b1b0 0x7f33bc06d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f33bc065430 msgr2=0x7f33bc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f33bc065430 0x7f33bc0678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f33cc004d80 tx=0x7f33cc00b040 comp rx=0 tx=0).stop 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 msgr2=0x7f33d010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d010e4f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f33c00043d0 tx=0x7f33c0009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 shutdown_connections 2024-06-13T01:24:19.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6810/2195816014,v1:172.21.15.132:6811/2195816014] conn(0x7f33bc06b1b0 0x7f33bc06d5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f33bc065430 0x7f33bc0678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33d0101b40 0x7f33d010e4f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 --2- 172.21.15.132:0/1936985005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33d01011a0 0x7f33d010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:19.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 >> 172.21.15.132:0/1936985005 conn(0x7f33d00f6d90 msgr2=0x7f33d00f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:19.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 shutdown_connections 2024-06-13T01:24:19.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:19.018+0000 7f33d7d19640 1 -- 172.21.15.132:0/1936985005 wait complete. 2024-06-13T01:24:19.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:18 smithi155 ceph-mon[35815]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:19.920 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:19.947 INFO:teuthology.orchestra.run.smithi132.stdout:60129542175 2024-06-13T01:24:19.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-13T01:24:20.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:20.819 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:20 smithi132 ceph-mon[29330]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:21.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:20 smithi155 ceph-mon[35815]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 -- 172.21.15.132:0/3503594405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c1022a0 msgr2=0x7f805c102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 --2- 172.21.15.132:0/3503594405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c1022a0 0x7f805c102670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f804c009c50 tx=0x7f804c004d10 comp rx=0 tx=0).stop 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 -- 172.21.15.132:0/3503594405 shutdown_connections 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 --2- 172.21.15.132:0/3503594405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f805c102c40 0x7f805c0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 --2- 172.21.15.132:0/3503594405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c1022a0 0x7f805c102670 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 -- 172.21.15.132:0/3503594405 >> 172.21.15.132:0/3503594405 conn(0x7f805c0f7e50 msgr2=0x7f805c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 -- 172.21.15.132:0/3503594405 shutdown_connections 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 -- 172.21.15.132:0/3503594405 wait complete. 2024-06-13T01:24:21.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 Processor -- start 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.526+0000 7f8061e69640 1 -- start start 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8061e69640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f805c1022a0 0x7f805c113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8061e69640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 0x7f805c1135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8061e69640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c113c30 con 0x7f805c102c40 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8061e69640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c10cfe0 con 0x7f805c1022a0 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 0x7f805c1135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 0x7f805c1135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59340/0 (socket says 172.21.15.132:59340) 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 -- 172.21.15.132:0/351063905 learned_addr learned my addr 172.21.15.132:0/351063905 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805b7fe640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f805c1022a0 0x7f805c113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 -- 172.21.15.132:0/351063905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f805c1022a0 msgr2=0x7f805c113060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f805c1022a0 0x7f805c113060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 -- 172.21.15.132:0/351063905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c0098b0 con 0x7f805c102c40 2024-06-13T01:24:21.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f805affd640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 0x7f805c1135a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f805000b790 tx=0x7f80500091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f805000c700 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f805000cd30 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c10d2c0 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.527+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8050013370 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.528+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c10d7e0 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.528+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f805c10dd20 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.529+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8050012070 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.529+0000 7f8043fff640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8044065430 0x7f80440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.529+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8050082040 con 0x7f805c102c40 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.529+0000 7f805b7fe640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8044065430 0x7f80440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.530+0000 7f8043fff640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] conn(0x7f804406b1b0 0x7f804406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:21.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.530+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 --> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f804406dc70 con 0x7f804406b1b0 2024-06-13T01:24:21.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.530+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8050084630 con 0x7f805c102c40 2024-06-13T01:24:21.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.530+0000 7f805bfff640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] conn(0x7f804406b1b0 0x7f804406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:21.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.530+0000 7f805b7fe640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8044065430 0x7f80440678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f804c009c20 tx=0x7f804c004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:21.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.530+0000 7f805bfff640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] conn(0x7f804406b1b0 0x7f804406d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:21.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.531+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== osd.2 v2:172.21.15.132:6818/2883035986 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f804406dc70 con 0x7f804406b1b0 2024-06-13T01:24:21.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.544+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 --> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f805c0f97e0 con 0x7f804406b1b0 2024-06-13T01:24:21.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.544+0000 7f8043fff640 1 -- 172.21.15.132:0/351063905 <== osd.2 v2:172.21.15.132:6818/2883035986 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f805c0f97e0 con 0x7f804406b1b0 2024-06-13T01:24:21.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] conn(0x7f804406b1b0 msgr2=0x7f804406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] conn(0x7f804406b1b0 0x7f804406d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8044065430 msgr2=0x7f80440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8044065430 0x7f80440678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f804c009c20 tx=0x7f804c004d10 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 msgr2=0x7f805c1135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 0x7f805c1135a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f805000b790 tx=0x7f80500091d0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 shutdown_connections 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6818/2883035986,v1:172.21.15.132:6819/2883035986] conn(0x7f804406b1b0 0x7f804406d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f8044065430 0x7f80440678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f805c102c40 0x7f805c1135a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 --2- 172.21.15.132:0/351063905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f805c1022a0 0x7f805c113060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:21.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 >> 172.21.15.132:0/351063905 conn(0x7f805c0f7e50 msgr2=0x7f805c0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:21.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 shutdown_connections 2024-06-13T01:24:21.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:21.545+0000 7f8061e69640 1 -- 172.21.15.132:0/351063905 wait complete. 2024-06-13T01:24:22.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/2940797647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae0101b70 msgr2=0x7f6ae00fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:22.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/2940797647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae0101b70 0x7f6ae00fb200 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6ad00048d0 tx=0x7f6ad0009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/2940797647 shutdown_connections 2024-06-13T01:24:22.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/2940797647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae0101b70 0x7f6ae00fb200 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/2940797647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ae01011d0 0x7f6ae01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/2940797647 >> 172.21.15.132:0/2940797647 conn(0x7f6ae00f6dc0 msgr2=0x7f6ae00f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/2940797647 shutdown_connections 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.618+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/2940797647 wait complete. 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6ae5d8a640 1 Processor -- start 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6ae5d8a640 1 -- start start 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6ae5d8a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 0x7f6ae0111f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6ae5d8a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ae0101b70 0x7f6ae0112440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6ae5d8a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0112a40 con 0x7f6ae01011d0 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6ae5d8a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae010be80 con 0x7f6ae0101b70 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adf7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 0x7f6ae0111f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adf7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 0x7f6ae0111f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49834/0 (socket says 172.21.15.132:49834) 2024-06-13T01:24:22.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adf7fe640 1 -- 172.21.15.132:0/405636160 learned_addr learned my addr 172.21.15.132:0/405636160 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adeffd640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ae0101b70 0x7f6ae0112440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adf7fe640 1 -- 172.21.15.132:0/405636160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ae0101b70 msgr2=0x7f6ae0112440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adf7fe640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ae0101b70 0x7f6ae0112440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.619+0000 7f6adf7fe640 1 -- 172.21.15.132:0/405636160 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0005040 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.620+0000 7f6adf7fe640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 0x7f6ae0111f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4002a50 tx=0x7f6ad4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.620+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad4007c30 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.620+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae010c160 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.620+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ad400f040 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.620+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad400dbe0 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.620+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae010c700 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.621+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ae010cbe0 con 0x7f6ae01011d0 2024-06-13T01:24:22.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.621+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6ad400dd40 con 0x7f6ae01011d0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.621+0000 7f6ac7fff640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ac8065430 0x7f6ac80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6adeffd640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ac8065430 0x7f6ac80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6ad4082470 con 0x7f6ae01011d0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6ac7fff640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] conn(0x7f6ac806b1b0 0x7f6ac806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 --> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac806dc70 con 0x7f6ac806b1b0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ad4084ac0 con 0x7f6ae01011d0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6adffff640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] conn(0x7f6ac806b1b0 0x7f6ac806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6adeffd640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ac8065430 0x7f6ac80678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ae010d7c0 tx=0x7f6ad0035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.622+0000 7f6adffff640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] conn(0x7f6ac806b1b0 0x7f6ac806d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.623+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== osd.0 v2:172.21.15.132:6802/3154009542 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6ac806dc70 con 0x7f6ac806b1b0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.635+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 --> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ae00fc940 con 0x7f6ac806b1b0 2024-06-13T01:24:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ac7fff640 1 -- 172.21.15.132:0/405636160 <== osd.0 v2:172.21.15.132:6802/3154009542 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ae00fc940 con 0x7f6ac806b1b0 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] conn(0x7f6ac806b1b0 msgr2=0x7f6ac806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] conn(0x7f6ac806b1b0 0x7f6ac806d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ac8065430 msgr2=0x7f6ac80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ac8065430 0x7f6ac80678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ae010d7c0 tx=0x7f6ad0035040 comp rx=0 tx=0).stop 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 msgr2=0x7f6ae0111f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 0x7f6ae0111f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4002a50 tx=0x7f6ad4004600 comp rx=0 tx=0).stop 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 shutdown_connections 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ac8065430 0x7f6ac80678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:6802/3154009542,v1:172.21.15.132:6803/3154009542] conn(0x7f6ac806b1b0 0x7f6ac806d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ae0101b70 0x7f6ae0112440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 --2- 172.21.15.132:0/405636160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6ae01011d0 0x7f6ae0111f00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 >> 172.21.15.132:0/405636160 conn(0x7f6ae00f6dc0 msgr2=0x7f6ae00f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 shutdown_connections 2024-06-13T01:24:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:22.636+0000 7f6ae5d8a640 1 -- 172.21.15.132:0/405636160 wait complete. 2024-06-13T01:24:22.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:22 smithi132 ceph-mon[29330]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:23.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:22 smithi155 ceph-mon[35815]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:23.336 INFO:teuthology.orchestra.run.smithi132.stdout:81604378650 2024-06-13T01:24:23.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3385324762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340650a0 msgr2=0x7fa434065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3385324762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340650a0 0x7fa434065470 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa4240048d0 tx=0x7fa424009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3385324762 shutdown_connections 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3385324762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4340fdeb0 0x7fa4340fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3385324762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340650a0 0x7fa434065470 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3385324762 >> 172.21.15.132:0/3385324762 conn(0x7fa434072110 msgr2=0x7fa434072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3385324762 shutdown_connections 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3385324762 wait complete. 2024-06-13T01:24:23.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.545+0000 7fa43a9f7640 1 Processor -- start 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa43a9f7640 1 -- start start 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa43a9f7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4340650a0 0x7fa434113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa43a9f7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 0x7fa434113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa43a9f7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa434113b60 con 0x7fa4340fdeb0 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa43a9f7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43410cfa0 con 0x7fa4340650a0 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 0x7fa434113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa4337fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 0x7fa434113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49848/0 (socket says 172.21.15.132:49848) 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa4337fe640 1 -- 172.21.15.132:0/3084106543 learned_addr learned my addr 172.21.15.132:0/3084106543 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:23.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa433fff640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4340650a0 0x7fa434113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa4337fe640 1 -- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4340650a0 msgr2=0x7fa434113020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa4337fe640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4340650a0 0x7fa434113020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.546+0000 7fa4337fe640 1 -- 172.21.15.132:0/3084106543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa424005040 con 0x7fa4340fdeb0 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.547+0000 7fa4337fe640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 0x7fa434113560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00eb70 tx=0x7fa41c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.547+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa41c007c30 con 0x7fa4340fdeb0 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.547+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43410d220 con 0x7fa4340fdeb0 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.547+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43410d7c0 con 0x7fa4340fdeb0 2024-06-13T01:24:23.549 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.547+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa41c00f040 con 0x7fa4340fdeb0 2024-06-13T01:24:23.550 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.547+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa41c0095b0 con 0x7fa4340fdeb0 2024-06-13T01:24:23.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.548+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa43410dcf0 con 0x7fa4340fdeb0 2024-06-13T01:24:23.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa41c009710 con 0x7fa4340fdeb0 2024-06-13T01:24:23.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa430ff9640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa420065430 0x7fa4200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:23.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa41c082170 con 0x7fa4340fdeb0 2024-06-13T01:24:23.551 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa433fff640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa420065430 0x7fa4200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa430ff9640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] conn(0x7fa42006b1b0 0x7fa42006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 --> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa42006dc70 con 0x7fa42006b1b0 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.549+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa41c0847c0 con 0x7fa4340fdeb0 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.550+0000 7fa433fff640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa420065430 0x7fa4200678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa424000c00 tx=0x7fa424035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.550+0000 7fa438f94640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] conn(0x7fa42006b1b0 0x7fa42006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.550+0000 7fa438f94640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] conn(0x7fa42006b1b0 0x7fa42006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:23.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.551+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== osd.5 v2:172.21.15.155:6816/410629247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa42006dc70 con 0x7fa42006b1b0 2024-06-13T01:24:23.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.564+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 --> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4340f9020 con 0x7fa42006b1b0 2024-06-13T01:24:23.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa430ff9640 1 -- 172.21.15.132:0/3084106543 <== osd.5 v2:172.21.15.155:6816/410629247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa4340f9020 con 0x7fa42006b1b0 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] conn(0x7fa42006b1b0 msgr2=0x7fa42006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] conn(0x7fa42006b1b0 0x7fa42006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa420065430 msgr2=0x7fa4200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa420065430 0x7fa4200678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa424000c00 tx=0x7fa424035040 comp rx=0 tx=0).stop 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 msgr2=0x7fa434113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 0x7fa434113560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00eb70 tx=0x7fa41c00b040 comp rx=0 tx=0).stop 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 shutdown_connections 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:6816/410629247,v1:172.21.15.155:6817/410629247] conn(0x7fa42006b1b0 0x7fa42006d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa420065430 0x7fa4200678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4340fdeb0 0x7fa434113560 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 --2- 172.21.15.132:0/3084106543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4340650a0 0x7fa434113020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.565+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 >> 172.21.15.132:0/3084106543 conn(0x7fa434072110 msgr2=0x7fa4340f8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.566+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 shutdown_connections 2024-06-13T01:24:23.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:23.566+0000 7fa43a9f7640 1 -- 172.21.15.132:0/3084106543 wait complete. 2024-06-13T01:24:24.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 -- 172.21.15.132:0/483384750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff644102c40 msgr2=0x7ff6440fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 --2- 172.21.15.132:0/483384750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff644102c40 0x7ff6440fc2d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff6380048d0 tx=0x7ff638009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 -- 172.21.15.132:0/483384750 shutdown_connections 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 --2- 172.21.15.132:0/483384750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff644102c40 0x7ff6440fc2d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 --2- 172.21.15.132:0/483384750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6441022a0 0x7ff644102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 -- 172.21.15.132:0/483384750 >> 172.21.15.132:0/483384750 conn(0x7ff6440f7e70 msgr2=0x7ff6440fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 -- 172.21.15.132:0/483384750 shutdown_connections 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.170+0000 7ff64a014640 1 -- 172.21.15.132:0/483384750 wait complete. 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff64a014640 1 Processor -- start 2024-06-13T01:24:24.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff64a014640 1 -- start start 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff64a014640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 0x7ff6441130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff64a014640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff644102c40 0x7ff644113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff64a014640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff644113c90 con 0x7ff6441022a0 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff64a014640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64410d040 con 0x7ff644102c40 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff6437fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 0x7ff6441130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff6437fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 0x7ff6441130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49866/0 (socket says 172.21.15.132:49866) 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.171+0000 7ff6437fe640 1 -- 172.21.15.132:0/406821626 learned_addr learned my addr 172.21.15.132:0/406821626 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff642ffd640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff644102c40 0x7ff644113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff6437fe640 1 -- 172.21.15.132:0/406821626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff644102c40 msgr2=0x7ff644113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff6437fe640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff644102c40 0x7ff644113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff6437fe640 1 -- 172.21.15.132:0/406821626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff638005040 con 0x7ff6441022a0 2024-06-13T01:24:24.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff642ffd640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff644102c40 0x7ff644113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff6437fe640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 0x7ff6441130c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff63400b790 tx=0x7ff6340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63400c700 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64410d320 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff63400cd30 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64410d840 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.172+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff634013370 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.173+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6440fda10 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.173+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff634012070 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.174+0000 7ff62bfff640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff62c065430 0x7ff62c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.174+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff634081f00 con 0x7ff6441022a0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.174+0000 7ff642ffd640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff62c065430 0x7ff62c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:24.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.174+0000 7ff642ffd640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff62c065430 0x7ff62c0678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff638009f80 tx=0x7ff638009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:24.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.178+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff6340503b0 con 0x7ff6441022a0 2024-06-13T01:24:24.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.346+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff6440fba90 con 0x7ff6441022a0 2024-06-13T01:24:24.346 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.346+0000 7ff62bfff640 1 -- 172.21.15.132:0/406821626 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff6340539d0 con 0x7ff6441022a0 2024-06-13T01:24:24.347 INFO:teuthology.orchestra.run.smithi132.stdout:146028888079 2024-06-13T01:24:24.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff62c065430 msgr2=0x7ff62c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:24.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff62c065430 0x7ff62c0678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff638009f80 tx=0x7ff638009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 msgr2=0x7ff6441130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:24.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 0x7ff6441130c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff63400b790 tx=0x7ff6340091d0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 shutdown_connections 2024-06-13T01:24:24.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff62c065430 0x7ff62c0678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff644102c40 0x7ff644113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 --2- 172.21.15.132:0/406821626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6441022a0 0x7ff6441130c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 >> 172.21.15.132:0/406821626 conn(0x7ff6440f7e70 msgr2=0x7ff6440f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:24.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.348+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 shutdown_connections 2024-06-13T01:24:24.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.349+0000 7ff64a014640 1 -- 172.21.15.132:0/406821626 wait complete. 2024-06-13T01:24:24.745 INFO:teuthology.orchestra.run.smithi132.stdout:167503724554 2024-06-13T01:24:24.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-13T01:24:24.955 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:24 smithi132 ceph-mon[29330]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:24.955 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:24 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/406821626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T01:24:24.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 -- 172.21.15.132:0/21646159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80fdeb0 msgr2=0x7f00f80fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:24.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/21646159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80fdeb0 0x7f00f80fe2e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f00e00048d0 tx=0x7f00e0009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 -- 172.21.15.132:0/21646159 shutdown_connections 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/21646159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80fdeb0 0x7f00f80fe2e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/21646159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00f80650a0 0x7f00f8065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 -- 172.21.15.132:0/21646159 >> 172.21.15.132:0/21646159 conn(0x7f00f8072110 msgr2=0x7f00f8072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 -- 172.21.15.132:0/21646159 shutdown_connections 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.955+0000 7f00fd2fd640 1 -- 172.21.15.132:0/21646159 wait complete. 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00fd2fd640 1 Processor -- start 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00fd2fd640 1 -- start start 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00fd2fd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 0x7f00f8113010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00fd2fd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00f80fdeb0 0x7f00f8113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00fd2fd640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8113b50 con 0x7f00f80650a0 2024-06-13T01:24:24.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00fd2fd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f810cf90 con 0x7f00f80fdeb0 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00f6ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 0x7f00f8113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00f6ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 0x7f00f8113010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49892/0 (socket says 172.21.15.132:49892) 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00f6ffd640 1 -- 172.21.15.132:0/932586175 learned_addr learned my addr 172.21.15.132:0/932586175 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00effff640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00f80fdeb0 0x7f00f8113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.956+0000 7f00f6ffd640 1 -- 172.21.15.132:0/932586175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00f80fdeb0 msgr2=0x7f00f8113550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00f6ffd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00f80fdeb0 0x7f00f8113550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00f6ffd640 1 -- 172.21.15.132:0/932586175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0005040 con 0x7f00f80650a0 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00f6ffd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 0x7f00f8113010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00e8002a50 tx=0x7f00e8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00e8007c30 con 0x7f00f80650a0 2024-06-13T01:24:24.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00e800f040 con 0x7f00f80650a0 2024-06-13T01:24:24.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f810d270 con 0x7f00f80650a0 2024-06-13T01:24:24.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00e800dad0 con 0x7f00f80650a0 2024-06-13T01:24:24.959 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.957+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f810d7e0 con 0x7f00f80650a0 2024-06-13T01:24:24.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.958+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f80fda50 con 0x7f00f80650a0 2024-06-13T01:24:24.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.958+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f00e800dd40 con 0x7f00f80650a0 2024-06-13T01:24:24.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.959+0000 7f00ef7fe640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f00e4065430 0x7f00e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:24.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.959+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f00e8081cd0 con 0x7f00f80650a0 2024-06-13T01:24:24.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.959+0000 7f00effff640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f00e4065430 0x7f00e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:24.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.959+0000 7f00effff640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f00e4065430 0x7f00e40678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009f80 tx=0x7f00e0035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:24.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:24.963+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f00e8050230 con 0x7f00f80650a0 2024-06-13T01:24:25.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:24 smithi155 ceph-mon[35815]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:25.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:24 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/406821626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-13T01:24:25.133 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.134+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f00f804b480 con 0x7f00f80650a0 2024-06-13T01:24:25.133 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.134+0000 7f00ef7fe640 1 -- 172.21.15.132:0/932586175 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f00e8053850 con 0x7f00f80650a0 2024-06-13T01:24:25.134 INFO:teuthology.orchestra.run.smithi132.stdout:111669149716 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f00e4065430 msgr2=0x7f00e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f00e4065430 0x7f00e40678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009f80 tx=0x7f00e0035040 comp rx=0 tx=0).stop 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 msgr2=0x7f00f8113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 0x7f00f8113010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f00e8002a50 tx=0x7f00e8004600 comp rx=0 tx=0).stop 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 shutdown_connections 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f00e4065430 0x7f00e40678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:25.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00f80fdeb0 0x7f00f8113550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:25.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 --2- 172.21.15.132:0/932586175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00f80650a0 0x7f00f8113010 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:25.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 >> 172.21.15.132:0/932586175 conn(0x7f00f8072110 msgr2=0x7f00f8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:25.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 shutdown_connections 2024-06-13T01:24:25.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:25.136+0000 7f00fd2fd640 1 -- 172.21.15.132:0/932586175 wait complete. 2024-06-13T01:24:25.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:25 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/932586175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T01:24:25.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149714 got 111669149716 for osd.3 2024-06-13T01:24:25.835 DEBUG:teuthology.parallel:result is None 2024-06-13T01:24:25.877 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:25 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/932586175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-13T01:24:26.271 INFO:teuthology.orchestra.run.smithi132.stdout:38654705700 2024-06-13T01:24:26.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-13T01:24:26.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888077 got 146028888079 for osd.4 2024-06-13T01:24:26.668 DEBUG:teuthology.parallel:result is None 2024-06-13T01:24:26.725 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:26.728 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:26.738 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:26.893 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:26 smithi132 ceph-mon[29330]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:27.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:26 smithi155 ceph-mon[35815]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:28.664 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:28.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:28 smithi132 ceph-mon[29330]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:29.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:28 smithi155 ceph-mon[35815]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:30.944 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:30 smithi155 ceph-mon[35815]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:30.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:30 smithi132 ceph-mon[29330]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:31.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2628041916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 msgr2=0x7f1ad40653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:31.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2628041916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad40653e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009c50 tx=0x7f1acc004d10 comp rx=0 tx=0).stop 2024-06-13T01:24:31.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2628041916 shutdown_connections 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2628041916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ad40fdea0 0x7f1ad40fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2628041916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad40653e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2628041916 >> 172.21.15.132:0/2628041916 conn(0x7f1ad4072110 msgr2=0x7f1ad4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2628041916 shutdown_connections 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2628041916 wait complete. 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 Processor -- start 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- start start 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad4112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ad40fdea0 0x7f1ad4113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad4113b30 con 0x7f1ad4065010 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.387+0000 7f1ad8d27640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad4113ca0 con 0x7f1ad40fdea0 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad4112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:31.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad4112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49910/0 (socket says 172.21.15.132:49910) 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 -- 172.21.15.132:0/2368154641 learned_addr learned my addr 172.21.15.132:0/2368154641 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad1d9b640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ad40fdea0 0x7f1ad4113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 -- 172.21.15.132:0/2368154641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ad40fdea0 msgr2=0x7f1ad4113530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ad40fdea0 0x7f1ad4113530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 -- 172.21.15.132:0/2368154641 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0098b0 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad259c640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad4112ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1acc004f80 tx=0x7f1acc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1acc007d50 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad410d0c0 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1acc032c00 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1acc03db90 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.388+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad410d580 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.389+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad40fda40 con 0x7f1ad4065010 2024-06-13T01:24:31.393 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.390+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1acc02f030 con 0x7f1ad4065010 2024-06-13T01:24:31.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.390+0000 7f1acaffd640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ac0065430 0x7f1ac00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:31.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.390+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1acc0a3650 con 0x7f1ad4065010 2024-06-13T01:24:31.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.390+0000 7f1ad1d9b640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ac0065430 0x7f1ac00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:31.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.391+0000 7f1ad1d9b640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ac0065430 0x7f1ac00678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ad410e8f0 tx=0x7f1abc00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:31.394 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.394+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1acc071b00 con 0x7f1ad4065010 2024-06-13T01:24:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.566+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1ad4072c20 con 0x7f1ad4065010 2024-06-13T01:24:31.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.567+0000 7f1acaffd640 1 -- 172.21.15.132:0/2368154641 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1acc075120 con 0x7f1ad4065010 2024-06-13T01:24:31.566 INFO:teuthology.orchestra.run.smithi132.stdout:81604378651 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ac0065430 msgr2=0x7f1ac00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ac0065430 0x7f1ac00678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ad410e8f0 tx=0x7f1abc00b040 comp rx=0 tx=0).stop 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 msgr2=0x7f1ad4112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad4112ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1acc004f80 tx=0x7f1acc004d10 comp rx=0 tx=0).stop 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 shutdown_connections 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1ac0065430 0x7f1ac00678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:31.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ad40fdea0 0x7f1ad4113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:31.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 --2- 172.21.15.132:0/2368154641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ad4065010 0x7f1ad4112ff0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:31.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 >> 172.21.15.132:0/2368154641 conn(0x7f1ad4072110 msgr2=0x7f1ad40f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:31.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 shutdown_connections 2024-06-13T01:24:31.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:31.569+0000 7f1ad8d27640 1 -- 172.21.15.132:0/2368154641 wait complete. 2024-06-13T01:24:31.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:31 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2368154641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T01:24:32.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:31 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2368154641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-13T01:24:32.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 -- 172.21.15.132:0/3375692753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a4102c40 msgr2=0x7f52a40fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 --2- 172.21.15.132:0/3375692753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a4102c40 0x7f52a40fc2d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f52980048d0 tx=0x7f5298009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 -- 172.21.15.132:0/3375692753 shutdown_connections 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 --2- 172.21.15.132:0/3375692753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a4102c40 0x7f52a40fc2d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 --2- 172.21.15.132:0/3375692753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a41022a0 0x7f52a4102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 -- 172.21.15.132:0/3375692753 >> 172.21.15.132:0/3375692753 conn(0x7f52a40f7e50 msgr2=0x7f52a40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 -- 172.21.15.132:0/3375692753 shutdown_connections 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.873+0000 7f52aa540640 1 -- 172.21.15.132:0/3375692753 wait complete. 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52aa540640 1 Processor -- start 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52aa540640 1 -- start start 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52aa540640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 0x7f52a41152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52aa540640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a4102c40 0x7f52a4115800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52aa540640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4115e90 con 0x7f52a41022a0 2024-06-13T01:24:32.879 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52aa540640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a410f240 con 0x7f52a4102c40 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52a3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 0x7f52a41152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52a3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 0x7f52a41152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32838/0 (socket says 172.21.15.132:32838) 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.874+0000 7f52a3fff640 1 -- 172.21.15.132:0/2040850733 learned_addr learned my addr 172.21.15.132:0/2040850733 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a37fe640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a4102c40 0x7f52a4115800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a3fff640 1 -- 172.21.15.132:0/2040850733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a4102c40 msgr2=0x7f52a4115800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a3fff640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a4102c40 0x7f52a4115800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a3fff640 1 -- 172.21.15.132:0/2040850733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5298005040 con 0x7f52a41022a0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a3fff640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 0x7f52a41152c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f529400b790 tx=0x7f52940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f529400c700 con 0x7f52a41022a0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a410f520 con 0x7f52a41022a0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f529400cd30 con 0x7f52a41022a0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5294013370 con 0x7f52a41022a0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.875+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a410fa40 con 0x7f52a41022a0 2024-06-13T01:24:32.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.876+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a40fda10 con 0x7f52a41022a0 2024-06-13T01:24:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.876+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5294012070 con 0x7f52a41022a0 2024-06-13T01:24:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.877+0000 7f52a0ff9640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f528c065430 0x7f528c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.877+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f52940820d0 con 0x7f52a41022a0 2024-06-13T01:24:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.877+0000 7f52a37fe640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f528c065430 0x7f528c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.877+0000 7f52a37fe640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f528c065430 0x7f528c0678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52a4110b80 tx=0x7f5298035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:32.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.881+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5294050580 con 0x7f52a41022a0 2024-06-13T01:24:32.939 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:32 smithi132 ceph-mon[29330]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:33.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3936453138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f24065030 msgr2=0x7f0f24065400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:33.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3936453138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f24065030 0x7f0f24065400 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0f180048d0 tx=0x7f0f18009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3936453138 shutdown_connections 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3936453138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f240fdea0 0x7f0f240fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3936453138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f24065030 0x7f0f24065400 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3936453138 >> 172.21.15.132:0/3936453138 conn(0x7f0f24072110 msgr2=0x7f0f24072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.756+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3936453138 shutdown_connections 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3936453138 wait complete. 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 Processor -- start 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 -- start start 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f24065030 0x7f0f24113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 0x7f0f24113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24113b60 con 0x7f0f240fdea0 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f2b17c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2410cff0 con 0x7f0f24065030 2024-06-13T01:24:33.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f23fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 0x7f0f24113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.757+0000 7f0f23fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 0x7f0f24113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32826/0 (socket says 172.21.15.132:32826) 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f23fff640 1 -- 172.21.15.132:0/3434464204 learned_addr learned my addr 172.21.15.132:0/3434464204 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f28f18640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f24065030 0x7f0f24113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f23fff640 1 -- 172.21.15.132:0/3434464204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f24065030 msgr2=0x7f0f24113020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f23fff640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f24065030 0x7f0f24113020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f23fff640 1 -- 172.21.15.132:0/3434464204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18005040 con 0x7f0f240fdea0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f23fff640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 0x7f0f24113560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0f10002a50 tx=0x7f0f10004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f10007c30 con 0x7f0f240fdea0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f1000f040 con 0x7f0f240fdea0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2410d2d0 con 0x7f0f240fdea0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f1000db60 con 0x7f0f240fdea0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.758+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2410d7f0 con 0x7f0f240fdea0 2024-06-13T01:24:33.016 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.759+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f240fda40 con 0x7f0f240fdea0 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.760+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0f1000dcc0 con 0x7f0f240fdea0 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.760+0000 7f0f217fa640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0f14065430 0x7f0f140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stdout:38654705702 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.760+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0f10082240 con 0x7f0f240fdea0 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.760+0000 7f0f28f18640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0f14065430 0x7f0f140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.761+0000 7f0f28f18640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0f14065430 0x7f0f140678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009f80 tx=0x7f0f18035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.764+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0f10050830 con 0x7f0f240fdea0 2024-06-13T01:24:33.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.939+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0f2410e6c0 con 0x7f0f240fdea0 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.940+0000 7f0f217fa640 1 -- 172.21.15.132:0/3434464204 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0f10053e50 con 0x7f0f240fdea0 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0f14065430 msgr2=0x7f0f140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0f14065430 0x7f0f140678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0f18009f80 tx=0x7f0f18035040 comp rx=0 tx=0).stop 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 msgr2=0x7f0f24113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 0x7f0f24113560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0f10002a50 tx=0x7f0f10004600 comp rx=0 tx=0).stop 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 shutdown_connections 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0f14065430 0x7f0f140678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0f240fdea0 0x7f0f24113560 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 --2- 172.21.15.132:0/3434464204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f24065030 0x7f0f24113020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 >> 172.21.15.132:0/3434464204 conn(0x7f0f24072110 msgr2=0x7f0f24105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 shutdown_connections 2024-06-13T01:24:33.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:32.942+0000 7f0f2b17c640 1 -- 172.21.15.132:0/3434464204 wait complete. 2024-06-13T01:24:33.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.060+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f52a404b4b0 con 0x7f52a41022a0 2024-06-13T01:24:33.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.060+0000 7f52a0ff9640 1 -- 172.21.15.132:0/2040850733 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5294053ba0 con 0x7f52a41022a0 2024-06-13T01:24:33.059 INFO:teuthology.orchestra.run.smithi132.stdout:167503724556 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f528c065430 msgr2=0x7f528c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f528c065430 0x7f528c0678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52a4110b80 tx=0x7f5298035040 comp rx=0 tx=0).stop 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 msgr2=0x7f52a41152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 0x7f52a41152c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f529400b790 tx=0x7f52940091d0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 shutdown_connections 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f528c065430 0x7f528c0678e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a4102c40 0x7f52a4115800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 --2- 172.21.15.132:0/2040850733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52a41022a0 0x7f52a41152c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 >> 172.21.15.132:0/2040850733 conn(0x7f52a40f7e50 msgr2=0x7f52a4071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 shutdown_connections 2024-06-13T01:24:33.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:33.062+0000 7f52aa540640 1 -- 172.21.15.132:0/2040850733 wait complete. 2024-06-13T01:24:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:32 smithi155 ceph-mon[35815]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:33.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:33 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3434464204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T01:24:33.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:33 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2040850733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T01:24:34.077 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378650 got 81604378651 for osd.2 2024-06-13T01:24:34.078 DEBUG:teuthology.parallel:result is None 2024-06-13T01:24:34.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:33 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3434464204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-13T01:24:34.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:33 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2040850733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-13T01:24:34.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 -- 172.21.15.132:0/70679863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90181023e0 msgr2=0x7f90181027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 --2- 172.21.15.132:0/70679863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90181023e0 0x7f90181027b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f90080048d0 tx=0x7f9008009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 -- 172.21.15.132:0/70679863 shutdown_connections 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 --2- 172.21.15.132:0/70679863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90180faa50 0x7f90180faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 --2- 172.21.15.132:0/70679863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90181023e0 0x7f90181027b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 -- 172.21.15.132:0/70679863 >> 172.21.15.132:0/70679863 conn(0x7f9018070ab0 msgr2=0x7f9018070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 -- 172.21.15.132:0/70679863 shutdown_connections 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.294+0000 7f901e54f640 1 -- 172.21.15.132:0/70679863 wait complete. 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f901e54f640 1 Processor -- start 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f901e54f640 1 -- start start 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f901e54f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 0x7f90181130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:34.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f901e54f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90181023e0 0x7f9018113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f901e54f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9018113c20 con 0x7f90180faa50 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f901e54f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901810d060 con 0x7f90181023e0 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f9017fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 0x7f90181130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f9017fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 0x7f90181130e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32852/0 (socket says 172.21.15.132:32852) 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.295+0000 7f9017fff640 1 -- 172.21.15.132:0/212091040 learned_addr learned my addr 172.21.15.132:0/212091040 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9017fff640 1 -- 172.21.15.132:0/212091040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90181023e0 msgr2=0x7f9018113620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9017fff640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90181023e0 0x7f9018113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9017fff640 1 -- 172.21.15.132:0/212091040 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008005040 con 0x7f90180faa50 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9017fff640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 0x7f90181130e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90080043d0 tx=0x7f9008009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9008038070 con 0x7f90180faa50 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901810d2e0 con 0x7f90180faa50 2024-06-13T01:24:34.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9008032970 con 0x7f90180faa50 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f900803da60 con 0x7f90180faa50 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.296+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901810d7a0 con 0x7f90180faa50 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.297+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9018065550 con 0x7f90180faa50 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.297+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9008030020 con 0x7f90180faa50 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.298+0000 7f9014ff9640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9000065430 0x7f90000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.298+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f90080a2630 con 0x7f90180faa50 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.298+0000 7f90177fe640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9000065430 0x7f90000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.298+0000 7f90177fe640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9000065430 0x7f90000678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f900c009af0 tx=0x7f900c009cd0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:34.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.302+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9008070ae0 con 0x7f90180faa50 2024-06-13T01:24:34.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.473+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9018109530 con 0x7f90180faa50 2024-06-13T01:24:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.474+0000 7f9014ff9640 1 -- 172.21.15.132:0/212091040 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9008074100 con 0x7f90180faa50 2024-06-13T01:24:34.474 INFO:teuthology.orchestra.run.smithi132.stdout:60129542178 2024-06-13T01:24:34.475 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9000065430 msgr2=0x7f90000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9000065430 0x7f90000678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f900c009af0 tx=0x7f900c009cd0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 msgr2=0x7f90181130e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 0x7f90181130e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90080043d0 tx=0x7f9008009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 shutdown_connections 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9000065430 0x7f90000678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90181023e0 0x7f9018113620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 --2- 172.21.15.132:0/212091040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90180faa50 0x7f90181130e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 >> 172.21.15.132:0/212091040 conn(0x7f9018070ab0 msgr2=0x7f90180f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 shutdown_connections 2024-06-13T01:24:34.476 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:34.476+0000 7f901e54f640 1 -- 172.21.15.132:0/212091040 wait complete. 2024-06-13T01:24:34.706 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-06-13T01:24:34.706 DEBUG:teuthology.parallel:result is None 2024-06-13T01:24:34.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:34 smithi132 ceph-mon[29330]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:34.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:34.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:34 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/212091040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T01:24:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:34 smithi155 ceph-mon[35815]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:34 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/212091040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-13T01:24:35.378 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542178 for osd.1 2024-06-13T01:24:35.378 DEBUG:teuthology.parallel:result is None 2024-06-13T01:24:35.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-06-13T01:24:35.850 DEBUG:teuthology.parallel:result is None 2024-06-13T01:24:35.850 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-13T01:24:35.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T01:24:36.114 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:36.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:36 smithi132 ceph-mon[29330]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:37.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:36 smithi155 ceph-mon[35815]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:37.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 -- 172.21.15.132:0/854441146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4100fa9e0 msgr2=0x7fd4100fae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:37.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 --2- 172.21.15.132:0/854441146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4100fa9e0 0x7fd4100fae50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd404009c50 tx=0x7fd404004d10 comp rx=0 tx=0).stop 2024-06-13T01:24:37.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 -- 172.21.15.132:0/854441146 shutdown_connections 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 --2- 172.21.15.132:0/854441146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4100fa9e0 0x7fd4100fae50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 --2- 172.21.15.132:0/854441146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd410102370 0x7fd410102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 -- 172.21.15.132:0/854441146 >> 172.21.15.132:0/854441146 conn(0x7fd410070a40 msgr2=0x7fd410070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 -- 172.21.15.132:0/854441146 shutdown_connections 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.619+0000 7fd4182f7640 1 -- 172.21.15.132:0/854441146 wait complete. 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd4182f7640 1 Processor -- start 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd4182f7640 1 -- start start 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd4182f7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4100fa9e0 0x7fd4101130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd4182f7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 0x7fd410113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd4182f7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410113cc0 con 0x7fd410102370 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd4182f7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41010d070 con 0x7fd4100fa9e0 2024-06-13T01:24:37.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd415892640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 0x7fd410113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd415892640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 0x7fd410113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32876/0 (socket says 172.21.15.132:32876) 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd415892640 1 -- 172.21.15.132:0/1378677676 learned_addr learned my addr 172.21.15.132:0/1378677676 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd415892640 1 -- 172.21.15.132:0/1378677676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4100fa9e0 msgr2=0x7fd4101130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd415892640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4100fa9e0 0x7fd4101130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.620+0000 7fd415892640 1 -- 172.21.15.132:0/1378677676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4040098b0 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.621+0000 7fd415892640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 0x7fd410113630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd404004f80 tx=0x7fd404004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.621+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd404008de0 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.621+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41010d2f0 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.621+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd404032c00 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.621+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd40403dbc0 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.621+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41010d810 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.622+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd410065530 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.622+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd40402f030 con 0x7fd410102370 2024-06-13T01:24:37.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.622+0000 7fd40effd640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd3f8065430 0x7fd3f80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:37.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.623+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd4040a3ae0 con 0x7fd410102370 2024-06-13T01:24:37.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.623+0000 7fd416093640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd3f8065430 0x7fd3f80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:37.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.623+0000 7fd416093640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd3f8065430 0x7fd3f80678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd4000097e0 tx=0x7fd400006d20 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:37.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.627+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd404071f90 con 0x7fd410102370 2024-06-13T01:24:37.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.790+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd410109540 con 0x7fd3f8065430 2024-06-13T01:24:37.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.791+0000 7fd40effd640 1 -- 172.21.15.132:0/1378677676 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27800 (secure 0 0 0) 0x7fd410109540 con 0x7fd3f8065430 2024-06-13T01:24:37.791 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:37.791 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-06-13T01:24:37.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd3f8065430 msgr2=0x7fd3f80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:37.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd3f8065430 0x7fd3f80678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd4000097e0 tx=0x7fd400006d20 comp rx=0 tx=0).stop 2024-06-13T01:24:37.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 msgr2=0x7fd410113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 0x7fd410113630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd404004f80 tx=0x7fd404004d10 comp rx=0 tx=0).stop 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 shutdown_connections 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd3f8065430 0x7fd3f80678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd410102370 0x7fd410113630 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 --2- 172.21.15.132:0/1378677676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4100fa9e0 0x7fd4101130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 >> 172.21.15.132:0/1378677676 conn(0x7fd410070a40 msgr2=0x7fd4100f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 shutdown_connections 2024-06-13T01:24:37.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:37.793+0000 7fd4182f7640 1 -- 172.21.15.132:0/1378677676 wait complete. 2024-06-13T01:24:38.462 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-06-13T01:24:37.371534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788016,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682896,"statfs":{"total":575970213888,"available":574139285504,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002240"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-13T01:23:44.738337+0000","last_change":"2024-06-13T01:23:18.662926+0000","last_active":"2024-06-13T01:23:44.738337+0000","last_peered":"2024-06-13T01:23:44.738337+0000","last_clean":"2024-06-13T01:23:44.738337+0000","last_became_active":"2024-06-13T01:23:18.662532+0000","last_became_peered":"2024-06-13T01:23:18.662532+0000","last_unstale":"2024-06-13T01:23:44.738337+0000","last_undegraded":"2024-06-13T01:23:44.738337+0000","last_fullsized":"2024-06-13T01:23:44.738337+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T01:22:29.849571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T01:22:29.849571+0000","last_clean_scrub_stamp":"2024-06-13T01:22:29.849571+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-14T06:54:25.925722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.31},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.32700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.374},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jun 13 01:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297708,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:23:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jun 13 01:23:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jun 13 01:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Jun 13 01:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32500000000000001},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.437},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34599999999999997},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46300000000000002,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.38}]},{"osd":2,"last update":"Thu Jun 13 01:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.37,"15min":0.374},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jun 13 01:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 13 01:24:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.40100000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.31,"15min":0.30399999999999999},"min":{"1min":0.23100000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jun 13 01:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42999999999999999,"5min":0.439,"15min":0.439},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35499999999999998,"15min":0.36099999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.313}]},{"osd":3,"last update":"Thu Jun 13 01:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.624},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jun 13 01:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T01:24:38.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph pg dump --format=json 2024-06-13T01:24:38.715 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:38.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:38 smithi132 ceph-mon[29330]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:38.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:38 smithi132 ceph-mon[29330]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T01:24:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:38 smithi155 ceph-mon[35815]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:38 smithi155 ceph-mon[35815]: from='client.14412 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 -- 172.21.15.132:0/2034709077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 msgr2=0x7f0d8c0fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/2034709077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c0fe2e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0048d0 tx=0x7f0d7c009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 -- 172.21.15.132:0/2034709077 shutdown_connections 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/2034709077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c0fe2e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/2034709077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d8c0650a0 0x7f0d8c065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 -- 172.21.15.132:0/2034709077 >> 172.21.15.132:0/2034709077 conn(0x7f0d8c072110 msgr2=0x7f0d8c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 -- 172.21.15.132:0/2034709077 shutdown_connections 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 -- 172.21.15.132:0/2034709077 wait complete. 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 Processor -- start 2024-06-13T01:24:40.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.403+0000 7f0d93b0d640 1 -- start start 2024-06-13T01:24:40.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d93b0d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d8c0650a0 0x7f0d8c113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:40.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d93b0d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:40.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d93b0d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c113c00 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d93b0d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c10cff0 con 0x7f0d8c0650a0 2024-06-13T01:24:40.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:32890/0 (socket says 172.21.15.132:32890) 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 -- 172.21.15.132:0/785257128 learned_addr learned my addr 172.21.15.132:0/785257128 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d918a9640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d8c0650a0 0x7f0d8c113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 -- 172.21.15.132:0/785257128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d8c0650a0 msgr2=0x7f0d8c113030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d8c0650a0 0x7f0d8c113030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 -- 172.21.15.132:0/785257128 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c005040 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d910a8640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c113570 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c003080 tx=0x7f0d7c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c038070 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d7c008ee0 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c03d870 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.404+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c10d270 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.405+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c10d730 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.405+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c0fda50 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.406+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0d7c030020 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.406+0000 7f0d827fc640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0d78065430 0x7f0d780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.406+0000 7f0d918a9640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0d78065430 0x7f0d780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.406+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0d7c0a2080 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.407+0000 7f0d918a9640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0d78065430 0x7f0d780678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0d88007540 tx=0x7f0d88009780 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:40.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.410+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0d7c070650 con 0x7f0d8c0fdeb0 2024-06-13T01:24:40.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.571+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 --> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d8c10e6e0 con 0x7f0d78065430 2024-06-13T01:24:40.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.572+0000 7f0d827fc640 1 -- 172.21.15.132:0/785257128 <== mgr.14210 v2:172.21.15.132:6800/2992514013 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27815 (secure 0 0 0) 0x7f0d8c10e6e0 con 0x7f0d78065430 2024-06-13T01:24:40.572 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:40.572 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-06-13T01:24:40.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.574+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0d78065430 msgr2=0x7f0d780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:40.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.574+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0d78065430 0x7f0d780678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0d88007540 tx=0x7f0d88009780 comp rx=0 tx=0).stop 2024-06-13T01:24:40.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.574+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 msgr2=0x7f0d8c113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:40.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.574+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c113570 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c003080 tx=0x7f0d7c009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.574+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 shutdown_connections 2024-06-13T01:24:40.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.574+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0d78065430 0x7f0d780678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.575+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d8c0fdeb0 0x7f0d8c113570 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.575+0000 7f0d93b0d640 1 --2- 172.21.15.132:0/785257128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d8c0650a0 0x7f0d8c113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:40.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.575+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 >> 172.21.15.132:0/785257128 conn(0x7f0d8c072110 msgr2=0x7f0d8c0f8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:40.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.575+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 shutdown_connections 2024-06-13T01:24:40.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:40.575+0000 7f0d93b0d640 1 -- 172.21.15.132:0/785257128 wait complete. 2024-06-13T01:24:40.936 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:40 smithi132 ceph-mon[29330]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:41.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:40 smithi155 ceph-mon[35815]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:41.354 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-06-13T01:24:39.371812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788016,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682896,"statfs":{"total":575970213888,"available":574139285504,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002034"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-13T01:23:44.738337+0000","last_change":"2024-06-13T01:23:18.662926+0000","last_active":"2024-06-13T01:23:44.738337+0000","last_peered":"2024-06-13T01:23:44.738337+0000","last_clean":"2024-06-13T01:23:44.738337+0000","last_became_active":"2024-06-13T01:23:18.662532+0000","last_became_peered":"2024-06-13T01:23:18.662532+0000","last_unstale":"2024-06-13T01:23:44.738337+0000","last_undegraded":"2024-06-13T01:23:44.738337+0000","last_fullsized":"2024-06-13T01:23:44.738337+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-13T01:22:29.849571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-13T01:22:29.849571+0000","last_clean_scrub_stamp":"2024-06-13T01:22:29.849571+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-14T06:54:25.925722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jun 13 01:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.31},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.32700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.374},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jun 13 01:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297708,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:23:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jun 13 01:23:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jun 13 01:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Jun 13 01:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 13 01:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32500000000000001},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34599999999999997},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46300000000000002,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jun 13 01:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.37,"15min":0.374},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jun 13 01:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jun 13 01:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.441}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 13 01:24:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.40100000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.31,"15min":0.30399999999999999},"min":{"1min":0.23100000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jun 13 01:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42999999999999999,"5min":0.439,"15min":0.439},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35499999999999998,"15min":0.36099999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.45400000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.313}]},{"osd":3,"last update":"Thu Jun 13 01:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.624},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.64000000000000001}]},{"osd":4,"last update":"Thu Jun 13 01:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-13T01:24:41.355 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-13T01:24:41.355 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-13T01:24:41.355 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-13T01:24:41.356 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:24:41.607 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:41.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:41 smithi132 ceph-mon[29330]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T01:24:42.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:41 smithi155 ceph-mon[35815]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-13T01:24:42.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:42 smithi132 ceph-mon[29330]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:43.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:42 smithi155 ceph-mon[35815]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 -- 172.21.15.132:0/2871696538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c0fa9c0 msgr2=0x7f765c0fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 --2- 172.21.15.132:0/2871696538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c0fa9c0 0x7f765c0fae30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f76440048d0 tx=0x7f7644009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 -- 172.21.15.132:0/2871696538 shutdown_connections 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 --2- 172.21.15.132:0/2871696538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c0fa9c0 0x7f765c0fae30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 --2- 172.21.15.132:0/2871696538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c102370 0x7f765c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 -- 172.21.15.132:0/2871696538 >> 172.21.15.132:0/2871696538 conn(0x7f765c0709e0 msgr2=0x7f765c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 -- 172.21.15.132:0/2871696538 shutdown_connections 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 -- 172.21.15.132:0/2871696538 wait complete. 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 Processor -- start 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.174+0000 7f7660eba640 1 -- start start 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7660eba640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0fa9c0 0x7f765c1130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7660eba640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 0x7f765c1135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7660eba640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c113c70 con 0x7f765c102370 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7660eba640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c10d020 con 0x7f765c0fa9c0 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 0x7f765c1135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 0x7f765c1135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59056/0 (socket says 172.21.15.132:59056) 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 -- 172.21.15.132:0/1234879367 learned_addr learned my addr 172.21.15.132:0/1234879367 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f765a59c640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0fa9c0 0x7f765c1130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 -- 172.21.15.132:0/1234879367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0fa9c0 msgr2=0x7f765c1130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:43.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0fa9c0 0x7f765c1130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 -- 172.21.15.132:0/1234879367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644005040 con 0x7f765c102370 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f765a59c640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0fa9c0 0x7f765c1130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7653fff640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 0x7f765c1135e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7644003080 tx=0x7f7644009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7644038070 con 0x7f765c102370 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c10d2a0 con 0x7f765c102370 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.175+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7644008ee0 con 0x7f765c102370 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.176+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764403d870 con 0x7f765c102370 2024-06-13T01:24:43.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.176+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c10d760 con 0x7f765c102370 2024-06-13T01:24:43.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.176+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c065570 con 0x7f765c102370 2024-06-13T01:24:43.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.177+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7644030020 con 0x7f765c102370 2024-06-13T01:24:43.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.177+0000 7f7652ffd640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7648065430 0x7f76480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:43.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.177+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f76440a2250 con 0x7f765c102370 2024-06-13T01:24:43.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.177+0000 7f765a59c640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7648065430 0x7f76480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:43.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.178+0000 7f765a59c640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7648065430 0x7f76480678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f764c0099f0 tx=0x7f764c008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:43.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.181+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7644070780 con 0x7f765c102370 2024-06-13T01:24:43.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.379+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f765c109560 con 0x7f765c102370 2024-06-13T01:24:43.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.379+0000 7f7652ffd640 1 -- 172.21.15.132:0/1234879367 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7644073da0 con 0x7f765c102370 2024-06-13T01:24:43.379 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:43.379 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7648065430 msgr2=0x7f76480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7648065430 0x7f76480678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f764c0099f0 tx=0x7f764c008040 comp rx=0 tx=0).stop 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 msgr2=0x7f765c1135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 0x7f765c1135e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7644003080 tx=0x7f7644009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 shutdown_connections 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7648065430 0x7f76480678e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f765c102370 0x7f765c1135e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 --2- 172.21.15.132:0/1234879367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0fa9c0 0x7f765c1130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 >> 172.21.15.132:0/1234879367 conn(0x7f765c0709e0 msgr2=0x7f765c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 shutdown_connections 2024-06-13T01:24:43.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:43.381+0000 7f7660eba640 1 -- 172.21.15.132:0/1234879367 wait complete. 2024-06-13T01:24:44.011 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:43 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1234879367' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:24:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:43 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1234879367' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:24:45.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:44 smithi155 ceph-mon[35815]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:45.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:44 smithi132 ceph-mon[29330]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:47.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:24:47.065 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:46 smithi132 ceph-mon[29330]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:47.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:46 smithi155 ceph-mon[35815]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:47.310 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:48.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:47 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:24:48.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:47 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:24:48.888 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 -- 172.21.15.132:0/3347389729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 msgr2=0x7f16a80fb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 --2- 172.21.15.132:0/3347389729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a80fb1d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f169c0048d0 tx=0x7f169c009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 -- 172.21.15.132:0/3347389729 shutdown_connections 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 --2- 172.21.15.132:0/3347389729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a80fb1d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 --2- 172.21.15.132:0/3347389729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16a81011a0 0x7f16a8101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 -- 172.21.15.132:0/3347389729 >> 172.21.15.132:0/3347389729 conn(0x7f16a80f6d50 msgr2=0x7f16a80f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 -- 172.21.15.132:0/3347389729 shutdown_connections 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.882+0000 7f16afca1640 1 -- 172.21.15.132:0/3347389729 wait complete. 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16afca1640 1 Processor -- start 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16afca1640 1 -- start start 2024-06-13T01:24:48.889 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16afca1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16a81011a0 0x7f16a8111ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16afca1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a8112430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16afca1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a8112a30 con 0x7f16a8101b40 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16afca1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a810be70 con 0x7f16a81011a0 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ad23c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a8112430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ad23c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a8112430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59088/0 (socket says 172.21.15.132:59088) 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ad23c640 1 -- 172.21.15.132:0/2919096271 learned_addr learned my addr 172.21.15.132:0/2919096271 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ada3d640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16a81011a0 0x7f16a8111ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ad23c640 1 -- 172.21.15.132:0/2919096271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16a81011a0 msgr2=0x7f16a8111ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ad23c640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16a81011a0 0x7f16a8111ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.883+0000 7f16ad23c640 1 -- 172.21.15.132:0/2919096271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c005040 con 0x7f16a8101b40 2024-06-13T01:24:48.890 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.884+0000 7f16ad23c640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a8112430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f169c003080 tx=0x7f169c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.884+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169c03a070 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.884+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f169c008ee0 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.884+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a810c0f0 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.884+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169c032a40 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.884+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a810c630 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.885+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a80fc910 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.885+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f169c030020 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.885+0000 7f169a7fc640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1694065430 0x7f16940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.886+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f169c0a43a0 con 0x7f16a8101b40 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.886+0000 7f16ada3d640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1694065430 0x7f16940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:48.891 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.886+0000 7f16ada3d640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1694065430 0x7f16940678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f16a40041d0 tx=0x7f16a4004330 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:48.892 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:48.890+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f169c0728d0 con 0x7f16a8101b40 2024-06-13T01:24:48.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:48 smithi132 ceph-mon[29330]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:48.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:24:48.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:24:48.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:24:49.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.088+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16a81082e0 con 0x7f16a8101b40 2024-06-13T01:24:49.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.089+0000 7f169a7fc640 1 -- 172.21.15.132:0/2919096271 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f169c075ef0 con 0x7f16a8101b40 2024-06-13T01:24:49.089 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:49.089 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1694065430 msgr2=0x7f16940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1694065430 0x7f16940678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f16a40041d0 tx=0x7f16a4004330 comp rx=0 tx=0).stop 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 msgr2=0x7f16a8112430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a8112430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f169c003080 tx=0x7f169c009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 shutdown_connections 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f1694065430 0x7f16940678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16a8101b40 0x7f16a8112430 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 --2- 172.21.15.132:0/2919096271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16a81011a0 0x7f16a8111ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 >> 172.21.15.132:0/2919096271 conn(0x7f16a80f6d50 msgr2=0x7f16a80f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 shutdown_connections 2024-06-13T01:24:49.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:49.091+0000 7f16afca1640 1 -- 172.21.15.132:0/2919096271 wait complete. 2024-06-13T01:24:49.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:48 smithi155 ceph-mon[35815]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:49.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:24:49.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:24:49.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:24:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:49 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2919096271' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:24:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:50.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:49 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2919096271' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:24:50.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:24:51.127 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:50 smithi155 ceph-mon[35815]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:51.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:50 smithi132 ceph-mon[29330]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:52.804 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:24:53.069 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:53.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:52 smithi155 ceph-mon[35815]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:53.192 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:52 smithi132 ceph-mon[29330]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 -- 172.21.15.132:0/559789225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8141022a0 msgr2=0x7fb814102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/559789225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8141022a0 0x7fb814102670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb8040048d0 tx=0x7fb804009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 -- 172.21.15.132:0/559789225 shutdown_connections 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/559789225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb814102c40 0x7fb8140fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/559789225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8141022a0 0x7fb814102670 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 -- 172.21.15.132:0/559789225 >> 172.21.15.132:0/559789225 conn(0x7fb8140f7e50 msgr2=0x7fb8140fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 -- 172.21.15.132:0/559789225 shutdown_connections 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 -- 172.21.15.132:0/559789225 wait complete. 2024-06-13T01:24:54.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.623+0000 7fb81a6b6640 1 Processor -- start 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb81a6b6640 1 -- start start 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb81a6b6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8141022a0 0x7fb814113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb81a6b6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 0x7fb814113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb81a6b6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb814113b40 con 0x7fb814102c40 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb81a6b6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81410cf80 con 0x7fb8141022a0 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 0x7fb814113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 0x7fb814113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55320/0 (socket says 172.21.15.132:55320) 2024-06-13T01:24:54.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 -- 172.21.15.132:0/3018877556 learned_addr learned my addr 172.21.15.132:0/3018877556 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb813fff640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8141022a0 0x7fb814113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 -- 172.21.15.132:0/3018877556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8141022a0 msgr2=0x7fb814113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8141022a0 0x7fb814113000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 -- 172.21.15.132:0/3018877556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804005040 con 0x7fb814102c40 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb813fff640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8141022a0 0x7fb814113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.624+0000 7fb8137fe640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 0x7fb814113540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c790 tx=0x7fb8080043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.625+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80800a820 con 0x7fb814102c40 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.625+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb80800ae50 con 0x7fb814102c40 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.625+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81410d260 con 0x7fb814102c40 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.625+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8080115e0 con 0x7fb814102c40 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.625+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81410d800 con 0x7fb814102c40 2024-06-13T01:24:54.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.626+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8140fda10 con 0x7fb814102c40 2024-06-13T01:24:54.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.626+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb808012400 con 0x7fb814102c40 2024-06-13T01:24:54.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.626+0000 7fb810ff9640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb7fc065430 0x7fb7fc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:24:54.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.627+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb808082690 con 0x7fb814102c40 2024-06-13T01:24:54.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.627+0000 7fb813fff640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb7fc065430 0x7fb7fc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:24:54.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.627+0000 7fb813fff640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb7fc065430 0x7fb7fc0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb804009f80 tx=0x7fb804009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:24:54.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.631+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb808050bc0 con 0x7fb814102c40 2024-06-13T01:24:54.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.832+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb814109400 con 0x7fb814102c40 2024-06-13T01:24:54.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.832+0000 7fb810ff9640 1 -- 172.21.15.132:0/3018877556 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb8080541e0 con 0x7fb814102c40 2024-06-13T01:24:54.832 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:24:54.832 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb7fc065430 msgr2=0x7fb7fc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb7fc065430 0x7fb7fc0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb804009f80 tx=0x7fb804009da0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 msgr2=0x7fb814113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 0x7fb814113540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c790 tx=0x7fb8080043c0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 shutdown_connections 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb7fc065430 0x7fb7fc0678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb814102c40 0x7fb814113540 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 --2- 172.21.15.132:0/3018877556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8141022a0 0x7fb814113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 >> 172.21.15.132:0/3018877556 conn(0x7fb8140f7e50 msgr2=0x7fb8140f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:24:54.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 shutdown_connections 2024-06-13T01:24:54.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:24:54.834+0000 7fb81a6b6640 1 -- 172.21.15.132:0/3018877556 wait complete. 2024-06-13T01:24:55.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:54 smithi155 ceph-mon[35815]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:55.205 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:54 smithi132 ceph-mon[29330]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:55.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:55 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3018877556' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:24:56.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:55 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3018877556' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:24:57.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:56 smithi155 ceph-mon[35815]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:57.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:56 smithi132 ceph-mon[29330]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:58.459 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:24:58.724 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:24:59.061 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:24:58 smithi132 ceph-mon[29330]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:24:59.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:24:58 smithi155 ceph-mon[35815]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:00.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/1937974206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f581011b0 msgr2=0x7f6f58101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:00.245 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/1937974206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f581011b0 0x7f6f58101580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6f480048d0 tx=0x7f6f48009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/1937974206 shutdown_connections 2024-06-13T01:25:00.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/1937974206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f58101b50 0x7f6f580fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/1937974206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f581011b0 0x7f6f58101580 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/1937974206 >> 172.21.15.132:0/1937974206 conn(0x7f6f580f6d60 msgr2=0x7f6f580f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:00.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/1937974206 shutdown_connections 2024-06-13T01:25:00.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/1937974206 wait complete. 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.245+0000 7f6f5d6dd640 1 Processor -- start 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f5d6dd640 1 -- start start 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f5d6dd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f581011b0 0x7f6f58111fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f5d6dd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 0x7f6f58112500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f5d6dd640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58112b90 con 0x7f6f58101b50 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f5d6dd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5810bf40 con 0x7f6f581011b0 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 0x7f6f58112500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:00.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 0x7f6f58112500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55328/0 (socket says 172.21.15.132:55328) 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 -- 172.21.15.132:0/3762003063 learned_addr learned my addr 172.21.15.132:0/3762003063 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f56ffd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f581011b0 0x7f6f58111fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 -- 172.21.15.132:0/3762003063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f581011b0 msgr2=0x7f6f58111fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f581011b0 0x7f6f58111fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 -- 172.21.15.132:0/3762003063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48005040 con 0x7f6f58101b50 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.246+0000 7f6f567fc640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 0x7f6f58112500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00b790 tx=0x7f6f4c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.247+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f4c00c700 con 0x7f6f58101b50 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.247+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5810c220 con 0x7f6f58101b50 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.247+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f4c00cd30 con 0x7f6f58101b50 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.247+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f4c013370 con 0x7f6f58101b50 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.247+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5810c740 con 0x7f6f58101b50 2024-06-13T01:25:00.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.248+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f580fc920 con 0x7f6f58101b50 2024-06-13T01:25:00.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.248+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6f4c012070 con 0x7f6f58101b50 2024-06-13T01:25:00.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.248+0000 7f6f3f7fe640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6f40065430 0x7f6f400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:00.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.249+0000 7f6f56ffd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6f40065430 0x7f6f400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:00.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.249+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6f4c081dd0 con 0x7f6f58101b50 2024-06-13T01:25:00.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.249+0000 7f6f56ffd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6f40065430 0x7f6f400678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f48032430 tx=0x7f6f48035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:00.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.253+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6f4c050280 con 0x7f6f58101b50 2024-06-13T01:25:00.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.453+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6f580fa9a0 con 0x7f6f58101b50 2024-06-13T01:25:00.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.454+0000 7f6f3f7fe640 1 -- 172.21.15.132:0/3762003063 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6f4c0538a0 con 0x7f6f58101b50 2024-06-13T01:25:00.454 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:00.454 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6f40065430 msgr2=0x7f6f400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6f40065430 0x7f6f400678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f48032430 tx=0x7f6f48035040 comp rx=0 tx=0).stop 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 msgr2=0x7f6f58112500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 0x7f6f58112500 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00b790 tx=0x7f6f4c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 shutdown_connections 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6f40065430 0x7f6f400678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f58101b50 0x7f6f58112500 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 --2- 172.21.15.132:0/3762003063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f581011b0 0x7f6f58111fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:00.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.455+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 >> 172.21.15.132:0/3762003063 conn(0x7f6f580f6d60 msgr2=0x7f6f580f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:00.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.456+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 shutdown_connections 2024-06-13T01:25:00.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:00.456+0000 7f6f5d6dd640 1 -- 172.21.15.132:0/3762003063 wait complete. 2024-06-13T01:25:01.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:00 smithi155 ceph-mon[35815]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:01.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:00 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3762003063' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:01.139 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:00 smithi132 ceph-mon[29330]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:01.140 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:00 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3762003063' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:03.125 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:02 smithi155 ceph-mon[35815]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:03.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:02 smithi132 ceph-mon[29330]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:04.168 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:04.432 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:04.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:04 smithi132 ceph-mon[29330]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:04.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:05.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:04 smithi155 ceph-mon[35815]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:05.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:06.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 -- 172.21.15.132:0/2907281471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101250 msgr2=0x7f4acc101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:06.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/2907281471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101250 0x7f4acc101620 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0048d0 tx=0x7f4abc009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 -- 172.21.15.132:0/2907281471 shutdown_connections 2024-06-13T01:25:06.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/2907281471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc101b60 0x7f4acc0fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/2907281471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101250 0x7f4acc101620 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 -- 172.21.15.132:0/2907281471 >> 172.21.15.132:0/2907281471 conn(0x7f4acc0709e0 msgr2=0x7f4acc070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 -- 172.21.15.132:0/2907281471 shutdown_connections 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.079+0000 7f4ad24f6640 1 -- 172.21.15.132:0/2907281471 wait complete. 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4ad24f6640 1 Processor -- start 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4ad24f6640 1 -- start start 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4ad24f6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc101250 0x7f4acc113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4ad24f6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 0x7f4acc113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:06.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4ad24f6640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc113d00 con 0x7f4acc101b60 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4ad24f6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc10d0b0 con 0x7f4acc101250 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4acb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 0x7f4acc113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4acb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 0x7f4acc113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57726/0 (socket says 172.21.15.132:57726) 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.080+0000 7f4acb7fe640 1 -- 172.21.15.132:0/633393014 learned_addr learned my addr 172.21.15.132:0/633393014 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4acbfff640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc101250 0x7f4acc113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4acb7fe640 1 -- 172.21.15.132:0/633393014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc101250 msgr2=0x7f4acc113130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4acb7fe640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc101250 0x7f4acc113130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4acb7fe640 1 -- 172.21.15.132:0/633393014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc005040 con 0x7f4acc101b60 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4acb7fe640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 0x7f4acc113670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400bae0 tx=0x7f4ac4009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac400cb10 con 0x7f4acc101b60 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc10d390 con 0x7f4acc101b60 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ac400cc70 con 0x7f4acc101b60 2024-06-13T01:25:06.083 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac4007770 con 0x7f4acc101b60 2024-06-13T01:25:06.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.081+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc10d8b0 con 0x7f4acc101b60 2024-06-13T01:25:06.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.082+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc0fcae0 con 0x7f4acc101b60 2024-06-13T01:25:06.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.082+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4ac40079e0 con 0x7f4acc101b60 2024-06-13T01:25:06.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.083+0000 7f4ac8ff9640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4ab4065430 0x7f4ab40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:06.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.083+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ac4082830 con 0x7f4acc101b60 2024-06-13T01:25:06.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.083+0000 7f4acbfff640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4ab4065430 0x7f4ab40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:06.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.083+0000 7f4acbfff640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4ab4065430 0x7f4ab40678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4abc032430 tx=0x7f4abc035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:06.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.087+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4ac404cdf0 con 0x7f4acc101b60 2024-06-13T01:25:06.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.284+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4acc108460 con 0x7f4acc101b60 2024-06-13T01:25:06.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.284+0000 7f4ac8ff9640 1 -- 172.21.15.132:0/633393014 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4ac4050d90 con 0x7f4acc101b60 2024-06-13T01:25:06.284 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:06.284 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4ab4065430 msgr2=0x7f4ab40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4ab4065430 0x7f4ab40678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4abc032430 tx=0x7f4abc035040 comp rx=0 tx=0).stop 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 msgr2=0x7f4acc113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 0x7f4acc113670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400bae0 tx=0x7f4ac4009810 comp rx=0 tx=0).stop 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 shutdown_connections 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4ab4065430 0x7f4ab40678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4acc101b60 0x7f4acc113670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 --2- 172.21.15.132:0/633393014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc101250 0x7f4acc113130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:06.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 >> 172.21.15.132:0/633393014 conn(0x7f4acc0709e0 msgr2=0x7f4acc1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:06.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 shutdown_connections 2024-06-13T01:25:06.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:06.286+0000 7f4ad24f6640 1 -- 172.21.15.132:0/633393014 wait complete. 2024-06-13T01:25:06.855 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:06 smithi132 ceph-mon[29330]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:06.855 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:06 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/633393014' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:07.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:06 smithi155 ceph-mon[35815]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:07.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:06 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/633393014' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:09.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:08 smithi155 ceph-mon[35815]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:09.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:08 smithi132 ceph-mon[29330]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:09.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:10.145 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:10.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:10 smithi132 ceph-mon[29330]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:11.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:10 smithi155 ceph-mon[35815]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 -- 172.21.15.132:0/473000717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 msgr2=0x7fa178102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 --2- 172.21.15.132:0/473000717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178102670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa1700048d0 tx=0x7fa170009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 -- 172.21.15.132:0/473000717 shutdown_connections 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 --2- 172.21.15.132:0/473000717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa178102c40 0x7fa1780fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 --2- 172.21.15.132:0/473000717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178102670 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 -- 172.21.15.132:0/473000717 >> 172.21.15.132:0/473000717 conn(0x7fa1780f7e70 msgr2=0x7fa1780fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 -- 172.21.15.132:0/473000717 shutdown_connections 2024-06-13T01:25:11.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 -- 172.21.15.132:0/473000717 wait complete. 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 Processor -- start 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 -- start start 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178113090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.811+0000 7fa17eedb640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa178102c40 0x7fa1781135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17eedb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa178113c60 con 0x7fa1781022a0 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17eedb640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17810d010 con 0x7fa178102c40 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178113090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178113090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57734/0 (socket says 172.21.15.132:57734) 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 -- 172.21.15.132:0/3031519804 learned_addr learned my addr 172.21.15.132:0/3031519804 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa177fff640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa178102c40 0x7fa1781135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 -- 172.21.15.132:0/3031519804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa178102c40 msgr2=0x7fa1781135d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa178102c40 0x7fa1781135d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:11.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 -- 172.21.15.132:0/3031519804 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa170005040 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17cc77640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178113090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa170003080 tx=0x7fa170009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa170038070 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa170008ee0 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17003d870 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17810d290 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.812+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17810d750 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.813+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1780fda10 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.814+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa170030020 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.814+0000 7fa1757fa640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa164065430 0x7fa1640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.814+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa1700a2080 con 0x7fa1781022a0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.814+0000 7fa177fff640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa164065430 0x7fa1640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:11.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.814+0000 7fa177fff640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa164065430 0x7fa1640678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa17810e950 tx=0x7fa168009780 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:11.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:11.818+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa170070650 con 0x7fa1781022a0 2024-06-13T01:25:12.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.015+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa17804b4b0 con 0x7fa1781022a0 2024-06-13T01:25:12.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.016+0000 7fa1757fa640 1 -- 172.21.15.132:0/3031519804 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa170073c70 con 0x7fa1781022a0 2024-06-13T01:25:12.015 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:12.016 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.017+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa164065430 msgr2=0x7fa1640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.017+0000 7fa17eedb640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa164065430 0x7fa1640678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa17810e950 tx=0x7fa168009780 comp rx=0 tx=0).stop 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.017+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 msgr2=0x7fa178113090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.017+0000 7fa17eedb640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178113090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa170003080 tx=0x7fa170009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 shutdown_connections 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa164065430 0x7fa1640678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:12.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa178102c40 0x7fa1781135d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:12.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 --2- 172.21.15.132:0/3031519804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1781022a0 0x7fa178113090 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:12.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 >> 172.21.15.132:0/3031519804 conn(0x7fa1780f7e70 msgr2=0x7fa1780f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:12.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 shutdown_connections 2024-06-13T01:25:12.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:12.018+0000 7fa17eedb640 1 -- 172.21.15.132:0/3031519804 wait complete. 2024-06-13T01:25:13.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:12 smithi155 ceph-mon[35815]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:13.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:12 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3031519804' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:13.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:12 smithi132 ceph-mon[29330]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:13.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:12 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3031519804' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:15.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:14 smithi155 ceph-mon[35815]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:15.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:14 smithi132 ceph-mon[29330]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:15.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:16.012 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:17.002 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:16 smithi132 ceph-mon[29330]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:17.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:16 smithi155 ceph-mon[35815]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 -- 172.21.15.132:0/3113329760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5658102c40 msgr2=0x7f56580fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 --2- 172.21.15.132:0/3113329760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5658102c40 0x7f56580fc2d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f56480048d0 tx=0x7f5648009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 -- 172.21.15.132:0/3113329760 shutdown_connections 2024-06-13T01:25:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 --2- 172.21.15.132:0/3113329760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5658102c40 0x7f56580fc2d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 --2- 172.21.15.132:0/3113329760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56581022a0 0x7f5658102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 -- 172.21.15.132:0/3113329760 >> 172.21.15.132:0/3113329760 conn(0x7f56580f7e50 msgr2=0x7f56580fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 -- 172.21.15.132:0/3113329760 shutdown_connections 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 -- 172.21.15.132:0/3113329760 wait complete. 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 Processor -- start 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.507+0000 7f565fd81640 1 -- start start 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565fd81640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 0x7f5658112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565fd81640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5658102c40 0x7f5658113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565fd81640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5658113b30 con 0x7f56581022a0 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565fd81640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565810cf70 con 0x7f5658102c40 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 0x7f5658112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 0x7f5658112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:57634/0 (socket says 172.21.15.132:57634) 2024-06-13T01:25:17.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 -- 172.21.15.132:0/198700071 learned_addr learned my addr 172.21.15.132:0/198700071 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 -- 172.21.15.132:0/198700071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5658102c40 msgr2=0x7f5658113530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5658102c40 0x7f5658113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 -- 172.21.15.132:0/198700071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5648005040 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565db1d640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 0x7f5658112ff0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f565400c810 tx=0x7f56540043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f565400a820 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f565400ae50 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565810d250 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56540115c0 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.508+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565810d7f0 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.509+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56580fda10 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.510+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f565400a980 con 0x7f56581022a0 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.510+0000 7f564e7fc640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5644065430 0x7f56440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:17.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.510+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5654082a60 con 0x7f56581022a0 2024-06-13T01:25:17.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.510+0000 7f565d31c640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5644065430 0x7f56440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:17.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.510+0000 7f565d31c640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5644065430 0x7f56440678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f564802a530 tx=0x7f5648009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:17.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.514+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5654050f10 con 0x7f56581022a0 2024-06-13T01:25:17.710 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.711+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f565810e690 con 0x7f56581022a0 2024-06-13T01:25:17.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.711+0000 7f564e7fc640 1 -- 172.21.15.132:0/198700071 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5654054530 con 0x7f56581022a0 2024-06-13T01:25:17.711 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:17.711 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:17.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.713+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5644065430 msgr2=0x7f56440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:17.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.713+0000 7f565fd81640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5644065430 0x7f56440678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f564802a530 tx=0x7f5648009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.713+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 msgr2=0x7f5658112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:17.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.713+0000 7f565fd81640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 0x7f5658112ff0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f565400c810 tx=0x7f56540043c0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.713+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 shutdown_connections 2024-06-13T01:25:17.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.714+0000 7f565fd81640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5644065430 0x7f56440678e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.714+0000 7f565fd81640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5658102c40 0x7f5658113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.714+0000 7f565fd81640 1 --2- 172.21.15.132:0/198700071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56581022a0 0x7f5658112ff0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:17.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.714+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 >> 172.21.15.132:0/198700071 conn(0x7f56580f7e50 msgr2=0x7f56580f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:17.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.714+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 shutdown_connections 2024-06-13T01:25:17.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:17.714+0000 7f565fd81640 1 -- 172.21.15.132:0/198700071 wait complete. 2024-06-13T01:25:17.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:17 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/198700071' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:18.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:17 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/198700071' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:19.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:18 smithi155 ceph-mon[35815]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:19.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:18 smithi132 ceph-mon[29330]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:20.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:20.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:21.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:20 smithi155 ceph-mon[35815]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:20 smithi132 ceph-mon[29330]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:21.326 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:21.590 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:22.937 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:22 smithi132 ceph-mon[29330]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:23.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:22 smithi155 ceph-mon[35815]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 -- 172.21.15.132:0/2983021743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fd2c0 msgr2=0x7f13f80fd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 --2- 172.21.15.132:0/2983021743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fd2c0 0x7f13f80fd690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f13f00048d0 tx=0x7f13f0009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 -- 172.21.15.132:0/2983021743 shutdown_connections 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 --2- 172.21.15.132:0/2983021743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fdc60 0x7f13f81062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 --2- 172.21.15.132:0/2983021743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fd2c0 0x7f13f80fd690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 -- 172.21.15.132:0/2983021743 >> 172.21.15.132:0/2983021743 conn(0x7f13f80f6d90 msgr2=0x7f13f80f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 -- 172.21.15.132:0/2983021743 shutdown_connections 2024-06-13T01:25:23.180 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 -- 172.21.15.132:0/2983021743 wait complete. 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 Processor -- start 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.174+0000 7f1400439640 1 -- start start 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f1400439640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fd2c0 0x7f13f81130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f1400439640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 0x7f13f8113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f1400439640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f8113c90 con 0x7f13f80fdc60 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f1400439640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f810d040 con 0x7f13f80fd2c0 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 0x7f13f8113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 0x7f13f8113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51498/0 (socket says 172.21.15.132:51498) 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 -- 172.21.15.132:0/4128850978 learned_addr learned my addr 172.21.15.132:0/4128850978 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fe1d5640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fd2c0 0x7f13f81130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 -- 172.21.15.132:0/4128850978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fd2c0 msgr2=0x7f13f81130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fd2c0 0x7f13f81130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 -- 172.21.15.132:0/4128850978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f0005040 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fe1d5640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fd2c0 0x7f13f81130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13fd9d4640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 0x7f13f8113600 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00b790 tx=0x7f13ec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13ec00c700 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13ec00cd30 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f810d320 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13ec013370 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.175+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f810d840 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.176+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f8065e50 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.177+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f13ec012070 con 0x7f13f80fdc60 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.177+0000 7f13f6ffd640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f13e0065430 0x7f13e00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.177+0000 7f13fe1d5640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f13e0065430 0x7f13e00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:23.182 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.177+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f13ec0820d0 con 0x7f13f80fdc60 2024-06-13T01:25:23.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.178+0000 7f13fe1d5640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f13e0065430 0x7f13e00678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009f80 tx=0x7f13f0035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:23.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.181+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f13ec050580 con 0x7f13f80fdc60 2024-06-13T01:25:23.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.385+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f13f81094a0 con 0x7f13f80fdc60 2024-06-13T01:25:23.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.386+0000 7f13f6ffd640 1 -- 172.21.15.132:0/4128850978 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f13ec053ba0 con 0x7f13f80fdc60 2024-06-13T01:25:23.385 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:23.386 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:23.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f13e0065430 msgr2=0x7f13e00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:23.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f13e0065430 0x7f13e00678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009f80 tx=0x7f13f0035040 comp rx=0 tx=0).stop 2024-06-13T01:25:23.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 msgr2=0x7f13f8113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:23.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 0x7f13f8113600 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00b790 tx=0x7f13ec0091d0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 shutdown_connections 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f13e0065430 0x7f13e00678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13f80fdc60 0x7f13f8113600 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 --2- 172.21.15.132:0/4128850978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13f80fd2c0 0x7f13f81130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 >> 172.21.15.132:0/4128850978 conn(0x7f13f80f6d90 msgr2=0x7f13f80f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 shutdown_connections 2024-06-13T01:25:23.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:23.388+0000 7f1400439640 1 -- 172.21.15.132:0/4128850978 wait complete. 2024-06-13T01:25:24.010 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:23 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/4128850978' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:24.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:23 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/4128850978' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:25.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:24 smithi155 ceph-mon[35815]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:25.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:24 smithi132 ceph-mon[29330]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:27.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:27.064 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:26 smithi132 ceph-mon[29330]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:27.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:26 smithi155 ceph-mon[35815]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:27.304 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 -- 172.21.15.132:0/2858347622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500650a0 msgr2=0x7fd950065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 --2- 172.21.15.132:0/2858347622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500650a0 0x7fd950065470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd9400048d0 tx=0x7fd940009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 -- 172.21.15.132:0/2858347622 shutdown_connections 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 --2- 172.21.15.132:0/2858347622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9500fdeb0 0x7fd9500fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 --2- 172.21.15.132:0/2858347622 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500650a0 0x7fd950065470 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 -- 172.21.15.132:0/2858347622 >> 172.21.15.132:0/2858347622 conn(0x7fd950072110 msgr2=0x7fd950072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 -- 172.21.15.132:0/2858347622 shutdown_connections 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.866+0000 7fd95609e640 1 -- 172.21.15.132:0/2858347622 wait complete. 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 Processor -- start 2024-06-13T01:25:28.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 -- start start 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9500650a0 0x7fd950113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 0x7fd9501135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950113c50 con 0x7fd9500fdeb0 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95010d000 con 0x7fd9500650a0 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 0x7fd9501135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 0x7fd9501135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51508/0 (socket says 172.21.15.132:51508) 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 -- 172.21.15.132:0/3291313360 learned_addr learned my addr 172.21.15.132:0/3291313360 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 -- 172.21.15.132:0/3291313360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9500650a0 msgr2=0x7fd950113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9500650a0 0x7fd950113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 -- 172.21.15.132:0/3291313360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940005040 con 0x7fd9500fdeb0 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd947fff640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 0x7fd9501135c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd93800b790 tx=0x7fd9380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:28.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd93800c700 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd93800cd30 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95010d2e0 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.867+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd938013370 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.868+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95010d800 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.868+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9500fda50 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.869+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd938012070 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.869+0000 7fd94cff9640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd93c065430 0x7fd93c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.869+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd9380820d0 con 0x7fd9500fdeb0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.869+0000 7fd94f7fe640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd93c065430 0x7fd93c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.870+0000 7fd94f7fe640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd93c065430 0x7fd93c0678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd940032430 tx=0x7fd940009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:28.883 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:28.873+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd938050580 con 0x7fd9500fdeb0 2024-06-13T01:25:28.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:28 smithi132 ceph-mon[29330]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:29.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.073+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd950072c20 con 0x7fd9500fdeb0 2024-06-13T01:25:29.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.074+0000 7fd94cff9640 1 -- 172.21.15.132:0/3291313360 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd950072c20 con 0x7fd9500fdeb0 2024-06-13T01:25:29.073 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:29.074 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd93c065430 msgr2=0x7fd93c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd93c065430 0x7fd93c0678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd940032430 tx=0x7fd940009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 msgr2=0x7fd9501135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 0x7fd9501135c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd93800b790 tx=0x7fd9380091d0 comp rx=0 tx=0).stop 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 shutdown_connections 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fd93c065430 0x7fd93c0678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9500fdeb0 0x7fd9501135c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 --2- 172.21.15.132:0/3291313360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9500650a0 0x7fd950113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 >> 172.21.15.132:0/3291313360 conn(0x7fd950072110 msgr2=0x7fd950100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 shutdown_connections 2024-06-13T01:25:29.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:29.076+0000 7fd95609e640 1 -- 172.21.15.132:0/3291313360 wait complete. 2024-06-13T01:25:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:28 smithi155 ceph-mon[35815]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:29.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:29 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3291313360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:30.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:29 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3291313360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:31.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:30 smithi155 ceph-mon[35815]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:31.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:30 smithi132 ceph-mon[29330]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:32.669 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:32.934 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:32.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:32 smithi132 ceph-mon[29330]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:32 smithi155 ceph-mon[35815]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:34.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 -- 172.21.15.132:0/2213516425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 msgr2=0x7f5848102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:34.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 --2- 172.21.15.132:0/2213516425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848102690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58380048d0 tx=0x7f5838009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 -- 172.21.15.132:0/2213516425 shutdown_connections 2024-06-13T01:25:34.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 --2- 172.21.15.132:0/2213516425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5848102c60 0x7f58480fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 --2- 172.21.15.132:0/2213516425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848102690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 -- 172.21.15.132:0/2213516425 >> 172.21.15.132:0/2213516425 conn(0x7f58480f7e50 msgr2=0x7f58480fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 -- 172.21.15.132:0/2213516425 shutdown_connections 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.573+0000 7f584df85640 1 -- 172.21.15.132:0/2213516425 wait complete. 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f584df85640 1 Processor -- start 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f584df85640 1 -- start start 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f584df85640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f584df85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5848102c60 0x7f5848113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f584df85640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5848113e70 con 0x7f58481022c0 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f584df85640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584810d2b0 con 0x7f5848102c60 2024-06-13T01:25:34.589 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f58477fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f58477fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46604/0 (socket says 172.21.15.132:46604) 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.574+0000 7f58477fe640 1 -- 172.21.15.132:0/2395323351 learned_addr learned my addr 172.21.15.132:0/2395323351 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f5846ffd640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5848102c60 0x7f5848113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f58477fe640 1 -- 172.21.15.132:0/2395323351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5848102c60 msgr2=0x7f5848113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f58477fe640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5848102c60 0x7f5848113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f58477fe640 1 -- 172.21.15.132:0/2395323351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838005040 con 0x7f58481022c0 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f58477fe640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848113330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5838003080 tx=0x7f5838009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5838038070 con 0x7f58481022c0 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584810d530 con 0x7f58481022c0 2024-06-13T01:25:34.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5838008ee0 con 0x7f58481022c0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f583803d870 con 0x7f58481022c0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.575+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584810da70 con 0x7f58481022c0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.576+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58480fda30 con 0x7f58481022c0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.576+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5838030020 con 0x7f58481022c0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.577+0000 7f582ffff640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5830065430 0x7f58300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.577+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58380a2170 con 0x7f58481022c0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.577+0000 7f5846ffd640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5830065430 0x7f58300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.577+0000 7f5846ffd640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5830065430 0x7f58300678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f584810ebf0 tx=0x7f583c009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:34.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.581+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5838070740 con 0x7f58481022c0 2024-06-13T01:25:34.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.781+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f584804b480 con 0x7f58481022c0 2024-06-13T01:25:34.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.781+0000 7f582ffff640 1 -- 172.21.15.132:0/2395323351 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5838073d60 con 0x7f58481022c0 2024-06-13T01:25:34.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:34.781 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5830065430 msgr2=0x7f58300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5830065430 0x7f58300678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f584810ebf0 tx=0x7f583c009040 comp rx=0 tx=0).stop 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 msgr2=0x7f5848113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848113330 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5838003080 tx=0x7f5838009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 shutdown_connections 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5830065430 0x7f58300678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5848102c60 0x7f5848113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 --2- 172.21.15.132:0/2395323351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58481022c0 0x7f5848113330 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:34.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 >> 172.21.15.132:0/2395323351 conn(0x7f58480f7e50 msgr2=0x7f58480f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:34.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.783+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 shutdown_connections 2024-06-13T01:25:34.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:34.784+0000 7f584df85640 1 -- 172.21.15.132:0/2395323351 wait complete. 2024-06-13T01:25:35.096 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:34 smithi132 ceph-mon[29330]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:35.097 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:35.097 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:34 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2395323351' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:34 smithi155 ceph-mon[35815]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:34 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2395323351' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:37.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:36 smithi155 ceph-mon[35815]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:37.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:36 smithi132 ceph-mon[29330]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:38.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:38.667 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:39.040 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:38 smithi132 ceph-mon[29330]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:38 smithi155 ceph-mon[35815]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 -- 172.21.15.132:0/54007413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04681011d0 msgr2=0x7f04681015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 --2- 172.21.15.132:0/54007413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04681011d0 0x7f04681015a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f04600048d0 tx=0x7f0460009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 -- 172.21.15.132:0/54007413 shutdown_connections 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 --2- 172.21.15.132:0/54007413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0468101b70 0x7f04680fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 --2- 172.21.15.132:0/54007413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04681011d0 0x7f04681015a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 -- 172.21.15.132:0/54007413 >> 172.21.15.132:0/54007413 conn(0x7f04680f6dc0 msgr2=0x7f04680f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 -- 172.21.15.132:0/54007413 shutdown_connections 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.288+0000 7f046f8ca640 1 -- 172.21.15.132:0/54007413 wait complete. 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046f8ca640 1 Processor -- start 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046f8ca640 1 -- start start 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046f8ca640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04681011d0 0x7f0468102050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:40.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046f8ca640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 0x7f0468102590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046f8ca640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0468102bb0 con 0x7f0468101b70 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046f8ca640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0468102d20 con 0x7f04681011d0 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 0x7f0468102590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 0x7f0468102590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46624/0 (socket says 172.21.15.132:46624) 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 -- 172.21.15.132:0/1885310337 learned_addr learned my addr 172.21.15.132:0/1885310337 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046d666640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04681011d0 0x7f0468102050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 -- 172.21.15.132:0/1885310337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04681011d0 msgr2=0x7f0468102050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04681011d0 0x7f0468102050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 -- 172.21.15.132:0/1885310337 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460005040 con 0x7f0468101b70 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046d666640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04681011d0 0x7f0468102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.289+0000 7f046ce65640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 0x7f0468102590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b790 tx=0x7f045c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.290+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f045c00c700 con 0x7f0468101b70 2024-06-13T01:25:40.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.290+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0468119400 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.290+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f045c00cd30 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.290+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f045c013370 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.290+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04681198c0 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.291+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04680fc940 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.291+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f045c012070 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.292+0000 7f0465ffb640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0450065430 0x7f04500678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.292+0000 7f046d666640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0450065430 0x7f04500678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.292+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f045c0820d0 con 0x7f0468101b70 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.292+0000 7f046d666640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0450065430 0x7f04500678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f80 tx=0x7f0460035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:40.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.296+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f045c050580 con 0x7f0468101b70 2024-06-13T01:25:40.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.495+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04680fa9c0 con 0x7f0468101b70 2024-06-13T01:25:40.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.496+0000 7f0465ffb640 1 -- 172.21.15.132:0/1885310337 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f045c053ba0 con 0x7f0468101b70 2024-06-13T01:25:40.496 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:40.497 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:40.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0450065430 msgr2=0x7f04500678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:40.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0450065430 0x7f04500678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f80 tx=0x7f0460035040 comp rx=0 tx=0).stop 2024-06-13T01:25:40.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 msgr2=0x7f0468102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:40.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 0x7f0468102590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b790 tx=0x7f045c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 shutdown_connections 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f0450065430 0x7f04500678e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0468101b70 0x7f0468102590 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 --2- 172.21.15.132:0/1885310337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04681011d0 0x7f0468102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 >> 172.21.15.132:0/1885310337 conn(0x7f04680f6dc0 msgr2=0x7f04680f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 shutdown_connections 2024-06-13T01:25:40.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:40.498+0000 7f046f8ca640 1 -- 172.21.15.132:0/1885310337 wait complete. 2024-06-13T01:25:41.083 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:40 smithi132 ceph-mon[29330]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:41.083 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:40 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1885310337' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:41.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:40 smithi155 ceph-mon[35815]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:41.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:40 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1885310337' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:43.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:42 smithi155 ceph-mon[35815]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:43.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:42 smithi132 ceph-mon[29330]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:44.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:44.366 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:45.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:44 smithi155 ceph-mon[35815]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:45.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:44 smithi132 ceph-mon[29330]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 -- 172.21.15.132:0/2665639447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2304101b70 msgr2=0x7f23040fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 --2- 172.21.15.132:0/2665639447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2304101b70 0x7f23040fb200 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f22f80048d0 tx=0x7f22f8009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 -- 172.21.15.132:0/2665639447 shutdown_connections 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 --2- 172.21.15.132:0/2665639447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2304101b70 0x7f23040fb200 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 --2- 172.21.15.132:0/2665639447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23041011d0 0x7f23041015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 -- 172.21.15.132:0/2665639447 >> 172.21.15.132:0/2665639447 conn(0x7f23040f6dc0 msgr2=0x7f23040f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 -- 172.21.15.132:0/2665639447 shutdown_connections 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 -- 172.21.15.132:0/2665639447 wait complete. 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.912+0000 7f23089c8640 1 Processor -- start 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f23089c8640 1 -- start start 2024-06-13T01:25:45.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f23089c8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 0x7f2304113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f23089c8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2304101b70 0x7f2304113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f23089c8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304113cd0 con 0x7f23041011d0 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f23089c8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230410d0c0 con 0x7f2304101b70 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 0x7f2304113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 0x7f2304113100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49832/0 (socket says 172.21.15.132:49832) 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 -- 172.21.15.132:0/1921067266 learned_addr learned my addr 172.21.15.132:0/1921067266 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 -- 172.21.15.132:0/1921067266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2304101b70 msgr2=0x7f2304113640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2304101b70 0x7f2304113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 -- 172.21.15.132:0/1921067266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f8005040 con 0x7f23041011d0 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f230259c640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 0x7f2304113100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00b790 tx=0x7f22fc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22fc00c700 con 0x7f23041011d0 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22fc00cd30 con 0x7f23041011d0 2024-06-13T01:25:45.920 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230410d3a0 con 0x7f23041011d0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.913+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22fc013370 con 0x7f23041011d0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.914+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230410d8c0 con 0x7f23041011d0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.915+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23040fc940 con 0x7f23041011d0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.915+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f22fc012070 con 0x7f23041011d0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.915+0000 7f22f2ffd640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f22ec065430 0x7f22ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.915+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f22fc081fe0 con 0x7f23041011d0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.915+0000 7f2301d9b640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f22ec065430 0x7f22ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.916+0000 7f2301d9b640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f22ec065430 0x7f22ec0678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f22f802a530 tx=0x7f22f8009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:45.921 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:45.919+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f22fc050510 con 0x7f23041011d0 2024-06-13T01:25:46.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.118+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f23041083f0 con 0x7f23041011d0 2024-06-13T01:25:46.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.118+0000 7f22f2ffd640 1 -- 172.21.15.132:0/1921067266 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f22fc04ee60 con 0x7f23041011d0 2024-06-13T01:25:46.118 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:46.119 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f22ec065430 msgr2=0x7f22ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f22ec065430 0x7f22ec0678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f22f802a530 tx=0x7f22f8009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 msgr2=0x7f2304113100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 0x7f2304113100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00b790 tx=0x7f22fc0091d0 comp rx=0 tx=0).stop 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 shutdown_connections 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f22ec065430 0x7f22ec0678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2304101b70 0x7f2304113640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 --2- 172.21.15.132:0/1921067266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23041011d0 0x7f2304113100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 >> 172.21.15.132:0/1921067266 conn(0x7f23040f6dc0 msgr2=0x7f23040657d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:46.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 shutdown_connections 2024-06-13T01:25:46.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:46.121+0000 7f23089c8640 1 -- 172.21.15.132:0/1921067266 wait complete. 2024-06-13T01:25:47.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:46 smithi155 ceph-mon[35815]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:47.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:46 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1921067266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:47.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:46 smithi132 ceph-mon[29330]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:47.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:46 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1921067266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:49.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:48 smithi155 ceph-mon[35815]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:49.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:48 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:25:49.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:48 smithi132 ceph-mon[29330]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:49.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:48 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:25:49.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:49.969 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:50.091 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:25:50.091 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:25:50.091 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:25:50.091 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:25:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:25:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:25:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:25:51.094 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:50 smithi132 ceph-mon[29330]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:51.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:50 smithi155 ceph-mon[35815]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 -- 172.21.15.132:0/448687169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c101b70 msgr2=0x7f9c7c0fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/448687169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c101b70 0x7f9c7c0fb200 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9c680048d0 tx=0x7f9c68009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 -- 172.21.15.132:0/448687169 shutdown_connections 2024-06-13T01:25:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/448687169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c101b70 0x7f9c7c0fb200 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/448687169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 -- 172.21.15.132:0/448687169 >> 172.21.15.132:0/448687169 conn(0x7f9c7c0f6d80 msgr2=0x7f9c7c0f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 -- 172.21.15.132:0/448687169 shutdown_connections 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.544+0000 7f9c82cd5640 1 -- 172.21.15.132:0/448687169 wait complete. 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.545+0000 7f9c82cd5640 1 Processor -- start 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.545+0000 7f9c82cd5640 1 -- start start 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c82cd5640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c82cd5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7c101b70 0x7f9c7c10e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c82cd5640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c10eb80 con 0x7f9c7c1011d0 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c82cd5640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c10ecf0 con 0x7f9c7c101b70 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c10dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49852/0 (socket says 172.21.15.132:49852) 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 -- 172.21.15.132:0/684076235 learned_addr learned my addr 172.21.15.132:0/684076235 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c7bfff640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7c101b70 0x7f9c7c10e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 -- 172.21.15.132:0/684076235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7c101b70 msgr2=0x7f9c7c10e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7c101b70 0x7f9c7c10e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 -- 172.21.15.132:0/684076235 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68005040 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c80a71640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c10dff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000c790 tx=0x7f9c700043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c7000a7a0 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c7000add0 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c117060 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c700115e0 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.546+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c117570 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.547+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7c0fc940 con 0x7f9c7c1011d0 2024-06-13T01:25:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.548+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9c70012400 con 0x7f9c7c1011d0 2024-06-13T01:25:51.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.548+0000 7f9c797fa640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9c6c065430 0x7f9c6c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:51.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.548+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9c70082510 con 0x7f9c7c1011d0 2024-06-13T01:25:51.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.548+0000 7f9c7bfff640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9c6c065430 0x7f9c6c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:51.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.549+0000 7f9c7bfff640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9c6c065430 0x7f9c6c0678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9c68032430 tx=0x7f9c68035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:51.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.552+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c70050b00 con 0x7f9c7c1011d0 2024-06-13T01:25:51.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.753+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c7c0fa9c0 con 0x7f9c7c1011d0 2024-06-13T01:25:51.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.753+0000 7f9c797fa640 1 -- 172.21.15.132:0/684076235 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9c70054120 con 0x7f9c7c1011d0 2024-06-13T01:25:51.753 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:51.753 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:51.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9c6c065430 msgr2=0x7f9c6c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:51.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9c6c065430 0x7f9c6c0678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9c68032430 tx=0x7f9c68035040 comp rx=0 tx=0).stop 2024-06-13T01:25:51.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 msgr2=0x7f9c7c10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:51.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c10dff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000c790 tx=0x7f9c700043c0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 shutdown_connections 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9c6c065430 0x7f9c6c0678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7c101b70 0x7f9c7c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 --2- 172.21.15.132:0/684076235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c7c1011d0 0x7f9c7c10dff0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 >> 172.21.15.132:0/684076235 conn(0x7f9c7c0f6d80 msgr2=0x7f9c7c0f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 shutdown_connections 2024-06-13T01:25:51.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:51.755+0000 7f9c82cd5640 1 -- 172.21.15.132:0/684076235 wait complete. 2024-06-13T01:25:52.095 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:51 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/684076235' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:52.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:51 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/684076235' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:53.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:52 smithi155 ceph-mon[35815]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:53.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:52 smithi132 ceph-mon[29330]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:55.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:54 smithi155 ceph-mon[35815]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:55.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:54 smithi132 ceph-mon[29330]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:55.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:25:55.695 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:25:57.092 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:56 smithi132 ceph-mon[29330]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:57.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:56 smithi155 ceph-mon[35815]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:57.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 -- 172.21.15.132:0/2881293740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fbe80 msgr2=0x7faeb40fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:57.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 --2- 172.21.15.132:0/2881293740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fbe80 0x7faeb40fcd80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae9c0048d0 tx=0x7fae9c009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 -- 172.21.15.132:0/2881293740 shutdown_connections 2024-06-13T01:25:57.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 --2- 172.21.15.132:0/2881293740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fbe80 0x7faeb40fcd80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 --2- 172.21.15.132:0/2881293740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fb570 0x7faeb40fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 -- 172.21.15.132:0/2881293740 >> 172.21.15.132:0/2881293740 conn(0x7faeb4072110 msgr2=0x7faeb4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 -- 172.21.15.132:0/2881293740 shutdown_connections 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 -- 172.21.15.132:0/2881293740 wait complete. 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 Processor -- start 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 -- start start 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.296+0000 7faeb884a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 0x7faeb4113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb884a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fbe80 0x7faeb4113590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb884a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4113c20 con 0x7faeb40fb570 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb884a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb410cfd0 con 0x7faeb40fbe80 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 0x7faeb4113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 0x7faeb4113050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56708/0 (socket says 172.21.15.132:56708) 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 -- 172.21.15.132:0/895828109 learned_addr learned my addr 172.21.15.132:0/895828109 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:25:57.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeabfff640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fbe80 0x7faeb4113590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 -- 172.21.15.132:0/895828109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fbe80 msgr2=0x7faeb4113590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fbe80 0x7faeb4113590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 -- 172.21.15.132:0/895828109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c005040 con 0x7faeb40fb570 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeabfff640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fbe80 0x7faeb4113590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb259c640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 0x7faeb4113050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faea400b790 tx=0x7faea40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea400c700 con 0x7faeb40fb570 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faea400cd30 con 0x7faeb40fb570 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb410d2b0 con 0x7faeb40fb570 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea4013370 con 0x7faeb40fb570 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.297+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb410d7d0 con 0x7faeb40fb570 2024-06-13T01:25:57.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.298+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb40fc920 con 0x7faeb40fb570 2024-06-13T01:25:57.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.299+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7faea4012070 con 0x7faeb40fb570 2024-06-13T01:25:57.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.299+0000 7faeaaffd640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faea0065430 0x7faea00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:25:57.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.299+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faea40820d0 con 0x7faeb40fb570 2024-06-13T01:25:57.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.299+0000 7faeabfff640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faea0065430 0x7faea00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:25:57.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.300+0000 7faeabfff640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faea0065430 0x7faea00678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009f80 tx=0x7fae9c009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:25:57.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.304+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faea4050580 con 0x7faeb40fb570 2024-06-13T01:25:57.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.503+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faeb404b4e0 con 0x7faeb40fb570 2024-06-13T01:25:57.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.503+0000 7faeaaffd640 1 -- 172.21.15.132:0/895828109 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7faea4053ba0 con 0x7faeb40fb570 2024-06-13T01:25:57.503 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:25:57.503 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faea0065430 msgr2=0x7faea00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faea0065430 0x7faea00678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009f80 tx=0x7fae9c009da0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 msgr2=0x7faeb4113050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 0x7faeb4113050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faea400b790 tx=0x7faea40091d0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 shutdown_connections 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faea0065430 0x7faea00678e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb40fbe80 0x7faeb4113590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 --2- 172.21.15.132:0/895828109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faeb40fb570 0x7faeb4113050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:25:57.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 >> 172.21.15.132:0/895828109 conn(0x7faeb4072110 msgr2=0x7faeb40f8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:25:57.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 shutdown_connections 2024-06-13T01:25:57.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:25:57.505+0000 7faeb884a640 1 -- 172.21.15.132:0/895828109 wait complete. 2024-06-13T01:25:58.104 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:57 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/895828109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:58.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:57 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/895828109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:25:59.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:25:58 smithi155 ceph-mon[35815]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:25:59.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:25:58 smithi132 ceph-mon[29330]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:01.107 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:01.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:00 smithi155 ceph-mon[35815]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:01.137 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:00 smithi132 ceph-mon[29330]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:01.374 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:03.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 -- 172.21.15.132:0/4095273939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0fdeb0 msgr2=0x7f525c0fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:03.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 --2- 172.21.15.132:0/4095273939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0fdeb0 0x7f525c0fe2e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f52440048d0 tx=0x7f5244009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.033 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 -- 172.21.15.132:0/4095273939 shutdown_connections 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 --2- 172.21.15.132:0/4095273939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0fdeb0 0x7f525c0fe2e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 --2- 172.21.15.132:0/4095273939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f525c0650a0 0x7f525c065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 -- 172.21.15.132:0/4095273939 >> 172.21.15.132:0/4095273939 conn(0x7f525c072110 msgr2=0x7f525c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 -- 172.21.15.132:0/4095273939 shutdown_connections 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 -- 172.21.15.132:0/4095273939 wait complete. 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.026+0000 7f5260cd0640 1 Processor -- start 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 -- start start 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 0x7f525c113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f525c0fdeb0 0x7f525c113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c113c00 con 0x7f525c0650a0 2024-06-13T01:26:03.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c10cff0 con 0x7f525c0fdeb0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 0x7f525c113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 0x7f525c113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:48500/0 (socket says 172.21.15.132:48500) 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 -- 172.21.15.132:0/2384083246 learned_addr learned my addr 172.21.15.132:0/2384083246 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 -- 172.21.15.132:0/2384083246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f525c0fdeb0 msgr2=0x7f525c113570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f525c0fdeb0 0x7f525c113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 -- 172.21.15.132:0/2384083246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5244005040 con 0x7f525c0650a0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f525a59c640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 0x7f525c113030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f525400bbb0 tx=0x7f5254009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f525400c9c0 con 0x7f525c0650a0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52540092e0 con 0x7f525c0650a0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c10d2d0 con 0x7f525c0650a0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5254007640 con 0x7f525c0650a0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.027+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c10d7f0 con 0x7f525c0650a0 2024-06-13T01:26:03.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.028+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c0fda50 con 0x7f525c0650a0 2024-06-13T01:26:03.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.029+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5254012070 con 0x7f525c0650a0 2024-06-13T01:26:03.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.029+0000 7f5252ffd640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5248065430 0x7f52480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:03.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.029+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5254082460 con 0x7f525c0650a0 2024-06-13T01:26:03.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.029+0000 7f5259d9b640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5248065430 0x7f52480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:03.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.030+0000 7f5259d9b640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5248065430 0x7f52480678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5244032430 tx=0x7f5244009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:03.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.033+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5254050990 con 0x7f525c0650a0 2024-06-13T01:26:03.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:02 smithi155 ceph-mon[35815]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:03.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:02 smithi132 ceph-mon[29330]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:03.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.238+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f525c072c20 con 0x7f525c0650a0 2024-06-13T01:26:03.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.238+0000 7f5252ffd640 1 -- 172.21.15.132:0/2384083246 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5254053fb0 con 0x7f525c0650a0 2024-06-13T01:26:03.238 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:03.238 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5248065430 msgr2=0x7f52480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5248065430 0x7f52480678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5244032430 tx=0x7f5244009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 msgr2=0x7f525c113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 0x7f525c113030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f525400bbb0 tx=0x7f5254009840 comp rx=0 tx=0).stop 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 shutdown_connections 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5248065430 0x7f52480678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f525c0fdeb0 0x7f525c113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 --2- 172.21.15.132:0/2384083246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f525c0650a0 0x7f525c113030 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 >> 172.21.15.132:0/2384083246 conn(0x7f525c072110 msgr2=0x7f525c0f8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 shutdown_connections 2024-06-13T01:26:03.240 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:03.240+0000 7f5260cd0640 1 -- 172.21.15.132:0/2384083246 wait complete. 2024-06-13T01:26:04.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:03 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2384083246' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:04.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:03 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2384083246' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:05.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:04 smithi132 ceph-mon[29330]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:05.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:05.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:04 smithi155 ceph-mon[35815]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:05.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:06.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:07.186 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:07.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:06 smithi132 ceph-mon[29330]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:07.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:06 smithi155 ceph-mon[35815]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:08.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 -- 172.21.15.132:0/208918473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e0065010 msgr2=0x7f03e00653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:08.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 --2- 172.21.15.132:0/208918473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e0065010 0x7f03e00653e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0048d0 tx=0x7f03dc009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:08.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 -- 172.21.15.132:0/208918473 shutdown_connections 2024-06-13T01:26:08.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 --2- 172.21.15.132:0/208918473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03e00fdea0 0x7f03e00fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 --2- 172.21.15.132:0/208918473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e0065010 0x7f03e00653e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 -- 172.21.15.132:0/208918473 >> 172.21.15.132:0/208918473 conn(0x7f03e0072110 msgr2=0x7f03e0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 -- 172.21.15.132:0/208918473 shutdown_connections 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 -- 172.21.15.132:0/208918473 wait complete. 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 Processor -- start 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.808+0000 7f03e774c640 1 -- start start 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e774c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03e0065010 0x7f03e0112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e774c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 0x7f03e0113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e774c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e0113b30 con 0x7f03e00fdea0 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e774c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e0113ca0 con 0x7f03e0065010 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 0x7f03e0113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 0x7f03e0113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:48524/0 (socket says 172.21.15.132:48524) 2024-06-13T01:26:08.825 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 -- 172.21.15.132:0/2846774636 learned_addr learned my addr 172.21.15.132:0/2846774636 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e54e8640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03e0065010 0x7f03e0112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 -- 172.21.15.132:0/2846774636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03e0065010 msgr2=0x7f03e0112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03e0065010 0x7f03e0112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 -- 172.21.15.132:0/2846774636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc005040 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e4ce7640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 0x7f03e0113530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f03d0002a50 tx=0x7f03d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03d0007c30 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03d000f040 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e010d120 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.809+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03d000db50 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.810+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e010d6c0 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.811+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e00fda40 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.811+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f03d001c400 con 0x7f03e00fdea0 2024-06-13T01:26:08.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.811+0000 7f03d5ffb640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f03cc065430 0x7f03cc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:08.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.811+0000 7f03e54e8640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f03cc065430 0x7f03cc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:08.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.811+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03d0081e00 con 0x7f03e00fdea0 2024-06-13T01:26:08.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.812+0000 7f03e54e8640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f03cc065430 0x7f03cc0678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0047f0 tx=0x7f03dc009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:08.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:08.816+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f03d00502b0 con 0x7f03e00fdea0 2024-06-13T01:26:09.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.015+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f03e0072c20 con 0x7f03e00fdea0 2024-06-13T01:26:09.015 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.015+0000 7f03d5ffb640 1 -- 172.21.15.132:0/2846774636 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f03d00538d0 con 0x7f03e00fdea0 2024-06-13T01:26:09.015 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:09.015 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f03cc065430 msgr2=0x7f03cc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f03cc065430 0x7f03cc0678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0047f0 tx=0x7f03dc009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 msgr2=0x7f03e0113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 0x7f03e0113530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f03d0002a50 tx=0x7f03d0004600 comp rx=0 tx=0).stop 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 shutdown_connections 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f03cc065430 0x7f03cc0678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03e00fdea0 0x7f03e0113530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:09.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 --2- 172.21.15.132:0/2846774636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03e0065010 0x7f03e0112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:09.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 >> 172.21.15.132:0/2846774636 conn(0x7f03e0072110 msgr2=0x7f03e00f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:09.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 shutdown_connections 2024-06-13T01:26:09.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:09.017+0000 7f03e774c640 1 -- 172.21.15.132:0/2846774636 wait complete. 2024-06-13T01:26:09.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:08 smithi132 ceph-mon[29330]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:09.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:08 smithi155 ceph-mon[35815]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:10.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:09 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2846774636' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:10.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:09 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2846774636' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:11.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:10 smithi132 ceph-mon[29330]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:11.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:10 smithi155 ceph-mon[35815]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:12.637 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:12.901 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:12.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:12 smithi132 ceph-mon[29330]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:13.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:12 smithi155 ceph-mon[35815]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:14.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 -- 172.21.15.132:0/2740084727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c102370 msgr2=0x7fb37c102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:14.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 --2- 172.21.15.132:0/2740084727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c102370 0x7fb37c102740 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb3780048d0 tx=0x7fb378009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 -- 172.21.15.132:0/2740084727 shutdown_connections 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 --2- 172.21.15.132:0/2740084727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c0fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 --2- 172.21.15.132:0/2740084727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c102370 0x7fb37c102740 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 -- 172.21.15.132:0/2740084727 >> 172.21.15.132:0/2740084727 conn(0x7fb37c0709e0 msgr2=0x7fb37c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 -- 172.21.15.132:0/2740084727 shutdown_connections 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 -- 172.21.15.132:0/2740084727 wait complete. 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.567+0000 7fb382abb640 1 Processor -- start 2024-06-13T01:26:14.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb382abb640 1 -- start start 2024-06-13T01:26:14.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb382abb640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:14.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb382abb640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb37c102370 0x7fb37c113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:14.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb382abb640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c113b60 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.570 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb382abb640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c10cfa0 con 0x7fb37c102370 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43484/0 (socket says 172.21.15.132:43484) 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 -- 172.21.15.132:0/1180465800 learned_addr learned my addr 172.21.15.132:0/1180465800 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb373fff640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb37c102370 0x7fb37c113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 -- 172.21.15.132:0/1180465800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb37c102370 msgr2=0x7fb37c113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb37c102370 0x7fb37c113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 -- 172.21.15.132:0/1180465800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378005040 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.568+0000 7fb380857640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c113020 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb3780043d0 tx=0x7fb378009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.569+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb378038070 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.569+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb378032970 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.569+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c10d220 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.569+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37803da60 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.569+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c10d760 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.570+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c065570 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.570+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb378030020 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.571+0000 7fb3717fa640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb364065430 0x7fb3640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.571+0000 7fb373fff640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb364065430 0x7fb3640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.571+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb3780a26d0 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.571+0000 7fb373fff640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb364065430 0x7fb3640678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb36c007fd0 tx=0x7fb36c009e90 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:14.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.575+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb378070b80 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.771+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb37c04b480 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.771+0000 7fb3717fa640 1 -- 172.21.15.132:0/1180465800 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb3780741a0 con 0x7fb37c0fa9c0 2024-06-13T01:26:14.771 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:14.772 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:14.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.773+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb364065430 msgr2=0x7fb3640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:14.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.773+0000 7fb382abb640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb364065430 0x7fb3640678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb36c007fd0 tx=0x7fb36c009e90 comp rx=0 tx=0).stop 2024-06-13T01:26:14.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.773+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 msgr2=0x7fb37c113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:14.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.773+0000 7fb382abb640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c113020 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb3780043d0 tx=0x7fb378009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.773+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 shutdown_connections 2024-06-13T01:26:14.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.774+0000 7fb382abb640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb364065430 0x7fb3640678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.774+0000 7fb382abb640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb37c102370 0x7fb37c113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.774+0000 7fb382abb640 1 --2- 172.21.15.132:0/1180465800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb37c0fa9c0 0x7fb37c113020 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:14.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.774+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 >> 172.21.15.132:0/1180465800 conn(0x7fb37c0709e0 msgr2=0x7fb37c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:14.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.774+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 shutdown_connections 2024-06-13T01:26:14.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:14.774+0000 7fb382abb640 1 -- 172.21.15.132:0/1180465800 wait complete. 2024-06-13T01:26:15.136 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:14 smithi132 ceph-mon[29330]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:15.136 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:14 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1180465800' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:15.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:14 smithi155 ceph-mon[35815]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:15.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:14 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1180465800' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:17.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:16 smithi132 ceph-mon[29330]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:17.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:16 smithi155 ceph-mon[35815]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:18.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:18.817 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:19.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:18 smithi132 ceph-mon[29330]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:19.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:18 smithi155 ceph-mon[35815]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:20.181 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:20.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- 172.21.15.132:0/2345770311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 msgr2=0x7ff32c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 --2- 172.21.15.132:0/2345770311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c101620 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff3280048d0 tx=0x7ff328009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- 172.21.15.132:0/2345770311 shutdown_connections 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 --2- 172.21.15.132:0/2345770311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff32c101b60 0x7ff32c0fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 --2- 172.21.15.132:0/2345770311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c101620 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- 172.21.15.132:0/2345770311 >> 172.21.15.132:0/2345770311 conn(0x7ff32c0709e0 msgr2=0x7ff32c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- 172.21.15.132:0/2345770311 shutdown_connections 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- 172.21.15.132:0/2345770311 wait complete. 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 Processor -- start 2024-06-13T01:26:20.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- start start 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c10e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff32c101b60 0x7ff32c10e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c10ed00 con 0x7ff32c101250 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff334520640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c10ee70 con 0x7ff32c101b60 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff3322bc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c10e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff3322bc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c10e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43506/0 (socket says 172.21.15.132:43506) 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.368+0000 7ff3322bc640 1 -- 172.21.15.132:0/654163170 learned_addr learned my addr 172.21.15.132:0/654163170 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff3322bc640 1 -- 172.21.15.132:0/654163170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff32c101b60 msgr2=0x7ff32c10e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff3322bc640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff32c101b60 0x7ff32c10e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff3322bc640 1 -- 172.21.15.132:0/654163170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff328005040 con 0x7ff32c101250 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff3322bc640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c10e0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff328004930 tx=0x7ff328009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:20.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff328038070 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff328032e30 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c117150 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff328033420 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.369+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c1175c0 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.370+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c0fcae0 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.370+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff328030020 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.371+0000 7ff322ffd640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff318065430 0x7ff3180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.371+0000 7ff331abb640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff318065430 0x7ff3180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.371+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff3280a2a80 con 0x7ff32c101250 2024-06-13T01:26:20.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.371+0000 7ff331abb640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff318065430 0x7ff3180678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0fb3c0 tx=0x7ff31c006d20 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:20.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:20.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.376+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff328070fb0 con 0x7ff32c101250 2024-06-13T01:26:20.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.571+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff32c0fb170 con 0x7ff32c101250 2024-06-13T01:26:20.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.572+0000 7ff322ffd640 1 -- 172.21.15.132:0/654163170 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff3280745d0 con 0x7ff32c101250 2024-06-13T01:26:20.573 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:20.573 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:20.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.573+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff318065430 msgr2=0x7ff3180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:20.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.573+0000 7ff334520640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff318065430 0x7ff3180678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0fb3c0 tx=0x7ff31c006d20 comp rx=0 tx=0).stop 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.573+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 msgr2=0x7ff32c10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.573+0000 7ff334520640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c10e0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff328004930 tx=0x7ff328009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 shutdown_connections 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff318065430 0x7ff3180678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff32c101b60 0x7ff32c10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 --2- 172.21.15.132:0/654163170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff32c101250 0x7ff32c10e0e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 >> 172.21.15.132:0/654163170 conn(0x7ff32c0709e0 msgr2=0x7ff32c0f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 shutdown_connections 2024-06-13T01:26:20.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:20.574+0000 7ff334520640 1 -- 172.21.15.132:0/654163170 wait complete. 2024-06-13T01:26:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:20 smithi132 ceph-mon[29330]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:20 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/654163170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:21.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:20 smithi155 ceph-mon[35815]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:21.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:20 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/654163170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:23.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:22 smithi132 ceph-mon[29330]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:23.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:22 smithi155 ceph-mon[35815]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:24.306 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:24.572 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:24.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:24 smithi132 ceph-mon[29330]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:25.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:24 smithi155 ceph-mon[35815]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:26.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.185+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/863703138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 msgr2=0x7f6cf41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:26.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.185+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/863703138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf41027b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40048d0 tx=0x7f6ce4009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.185+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/863703138 shutdown_connections 2024-06-13T01:26:26.187 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/863703138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cf40faa50 0x7f6cf40faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/863703138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf41027b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/863703138 >> 172.21.15.132:0/863703138 conn(0x7f6cf4070ab0 msgr2=0x7f6cf4070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/863703138 shutdown_connections 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/863703138 wait complete. 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 Processor -- start 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 -- start start 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cf40faa50 0x7f6cf406b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf406b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf406f230 con 0x7f6cf41023e0 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.186+0000 7f6cfa5fa640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf406f3a0 con 0x7f6cf40faa50 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf406b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf406b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33202/0 (socket says 172.21.15.132:33202) 2024-06-13T01:26:26.188 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 -- 172.21.15.132:0/2343601819 learned_addr learned my addr 172.21.15.132:0/2343601819 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf3fff640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cf40faa50 0x7f6cf406b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 -- 172.21.15.132:0/2343601819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cf40faa50 msgr2=0x7f6cf406b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cf40faa50 0x7f6cf406b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 -- 172.21.15.132:0/2343601819 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4005040 con 0x7f6cf41023e0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf37fe640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf406b650 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00c830 tx=0x7f6cdc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cdc00a7d0 con 0x7f6cf41023e0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf406bb90 con 0x7f6cf41023e0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cdc00ae00 con 0x7f6cf41023e0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cdc0113a0 con 0x7f6cf41023e0 2024-06-13T01:26:26.189 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.187+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf411fe30 con 0x7f6cf41023e0 2024-06-13T01:26:26.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.188+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf4065550 con 0x7f6cf41023e0 2024-06-13T01:26:26.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.189+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6cdc012400 con 0x7f6cf41023e0 2024-06-13T01:26:26.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.189+0000 7f6cf0ff9640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ce0065480 0x7f6ce0067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:26.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.189+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6cdc0825e0 con 0x7f6cf41023e0 2024-06-13T01:26:26.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.189+0000 7f6cf3fff640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ce0065480 0x7f6ce0067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:26.190 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.190+0000 7f6cf3fff640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ce0065480 0x7f6ce0067930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4000c00 tx=0x7f6ce4035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:26.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.193+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6cdc050b10 con 0x7f6cf41023e0 2024-06-13T01:26:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.389+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6cf40749d0 con 0x7f6cf41023e0 2024-06-13T01:26:26.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.389+0000 7f6cf0ff9640 1 -- 172.21.15.132:0/2343601819 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6cdc054130 con 0x7f6cf41023e0 2024-06-13T01:26:26.389 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:26.389 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ce0065480 msgr2=0x7f6ce0067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ce0065480 0x7f6ce0067930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4000c00 tx=0x7f6ce4035040 comp rx=0 tx=0).stop 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 msgr2=0x7f6cf406b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf406b650 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00c830 tx=0x7f6cdc004430 comp rx=0 tx=0).stop 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 shutdown_connections 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6ce0065480 0x7f6ce0067930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cf41023e0 0x7f6cf406b650 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 --2- 172.21.15.132:0/2343601819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cf40faa50 0x7f6cf406b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 >> 172.21.15.132:0/2343601819 conn(0x7f6cf4070ab0 msgr2=0x7f6cf40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:26.391 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 shutdown_connections 2024-06-13T01:26:26.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:26.391+0000 7f6cfa5fa640 1 -- 172.21.15.132:0/2343601819 wait complete. 2024-06-13T01:26:27.189 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:26 smithi132 ceph-mon[29330]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:27.189 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:26 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2343601819' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:27.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:26 smithi155 ceph-mon[35815]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:27.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:26 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2343601819' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:29.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:28 smithi132 ceph-mon[29330]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:29.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:28 smithi155 ceph-mon[35815]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:30.216 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:30.479 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:31.180 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:30 smithi155 ceph-mon[35815]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:31.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:30 smithi132 ceph-mon[29330]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:32.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 -- 172.21.15.132:0/348023073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 msgr2=0x7ff5c4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:32.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/348023073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4102690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40048d0 tx=0x7ff5b4009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 -- 172.21.15.132:0/348023073 shutdown_connections 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/348023073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 0x7ff5c40fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/348023073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4102690 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 -- 172.21.15.132:0/348023073 >> 172.21.15.132:0/348023073 conn(0x7ff5c40f7e50 msgr2=0x7ff5c40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 -- 172.21.15.132:0/348023073 shutdown_connections 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.050+0000 7ff5caa0f640 1 -- 172.21.15.132:0/348023073 wait complete. 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5caa0f640 1 Processor -- start 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5caa0f640 1 -- start start 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5caa0f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5caa0f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 0x7ff5c4113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5caa0f640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4113e70 con 0x7ff5c41022c0 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5caa0f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c410d2b0 con 0x7ff5c4102c60 2024-06-13T01:26:32.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c3fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37426/0 (socket says 172.21.15.132:37426) 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c3fff640 1 -- 172.21.15.132:0/821630221 learned_addr learned my addr 172.21.15.132:0/821630221 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c37fe640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 0x7ff5c4113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c3fff640 1 -- 172.21.15.132:0/821630221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 msgr2=0x7ff5c4113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c3fff640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 0x7ff5c4113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c3fff640 1 -- 172.21.15.132:0/821630221 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b4005040 con 0x7ff5c41022c0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.051+0000 7ff5c37fe640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 0x7ff5c4113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.052+0000 7ff5c3fff640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4113330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4003080 tx=0x7ff5b4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.052+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5b4038070 con 0x7ff5c41022c0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.052+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5b4032970 con 0x7ff5c41022c0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.052+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c410d530 con 0x7ff5c41022c0 2024-06-13T01:26:32.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.052+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5b403d960 con 0x7ff5c41022c0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.052+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c410da70 con 0x7ff5c41022c0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.053+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c40fda30 con 0x7ff5c41022c0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.053+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff5b4030020 con 0x7ff5c41022c0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.054+0000 7ff5c0ff9640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff5ac065430 0x7ff5ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.054+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff5b40a2500 con 0x7ff5c41022c0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.054+0000 7ff5c37fe640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff5ac065430 0x7ff5ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.054+0000 7ff5c37fe640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff5ac065430 0x7ff5ac0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff5c410ebf0 tx=0x7ff5b800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:32.061 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.058+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff5b40709b0 con 0x7ff5c41022c0 2024-06-13T01:26:32.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.260+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff5c40fbab0 con 0x7ff5c41022c0 2024-06-13T01:26:32.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.260+0000 7ff5c0ff9640 1 -- 172.21.15.132:0/821630221 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff5b4044300 con 0x7ff5c41022c0 2024-06-13T01:26:32.260 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:32.260 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:32.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff5ac065430 msgr2=0x7ff5ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:32.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff5ac065430 0x7ff5ac0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff5c410ebf0 tx=0x7ff5b800b040 comp rx=0 tx=0).stop 2024-06-13T01:26:32.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 msgr2=0x7ff5c4113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4113330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4003080 tx=0x7ff5b4009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 shutdown_connections 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff5ac065430 0x7ff5ac0678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102c60 0x7ff5c4113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 --2- 172.21.15.132:0/821630221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5c41022c0 0x7ff5c4113330 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 >> 172.21.15.132:0/821630221 conn(0x7ff5c40f7e50 msgr2=0x7ff5c40f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 shutdown_connections 2024-06-13T01:26:32.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:32.262+0000 7ff5caa0f640 1 -- 172.21.15.132:0/821630221 wait complete. 2024-06-13T01:26:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:32 smithi132 ceph-mon[29330]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:33.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:32 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/821630221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:32 smithi155 ceph-mon[35815]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:33.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:32 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/821630221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:35.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:34 smithi132 ceph-mon[29330]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:35.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:35.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:34 smithi155 ceph-mon[35815]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:35.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:35.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:36.157 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:37.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:36 smithi132 ceph-mon[29330]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:37.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:36 smithi155 ceph-mon[35815]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/52203183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c1011b0 msgr2=0x7f7a9c101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/52203183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c1011b0 0x7f7a9c101580 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7a900048d0 tx=0x7f7a90009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/52203183 shutdown_connections 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/52203183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a9c101b50 0x7f7a9c0fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/52203183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c1011b0 0x7f7a9c101580 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/52203183 >> 172.21.15.132:0/52203183 conn(0x7f7a9c0f6d60 msgr2=0x7f7a9c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/52203183 shutdown_connections 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/52203183 wait complete. 2024-06-13T01:26:37.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.767+0000 7f7aa2fb7640 1 Processor -- start 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7aa2fb7640 1 -- start start 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7aa2fb7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a9c1011b0 0x7f7a9c111fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7aa2fb7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 0x7f7a9c112500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7aa2fb7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c112b90 con 0x7f7a9c101b50 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7aa2fb7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c10bf80 con 0x7f7a9c1011b0 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7a9bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 0x7f7a9c112500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7a9bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 0x7f7a9c112500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:37450/0 (socket says 172.21.15.132:37450) 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7a9bfff640 1 -- 172.21.15.132:0/2908268518 learned_addr learned my addr 172.21.15.132:0/2908268518 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7aa0d53640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a9c1011b0 0x7f7a9c111fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7a9bfff640 1 -- 172.21.15.132:0/2908268518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a9c1011b0 msgr2=0x7f7a9c111fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:37.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7a9bfff640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a9c1011b0 0x7f7a9c111fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.768+0000 7f7a9bfff640 1 -- 172.21.15.132:0/2908268518 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90005040 con 0x7f7a9c101b50 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.769+0000 7f7a9bfff640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 0x7f7a9c112500 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800b790 tx=0x7f7a880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.769+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a8800c700 con 0x7f7a9c101b50 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.769+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a8800cd30 con 0x7f7a9c101b50 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.769+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a88013370 con 0x7f7a9c101b50 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.769+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c10c260 con 0x7f7a9c101b50 2024-06-13T01:26:37.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.769+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c10c780 con 0x7f7a9c101b50 2024-06-13T01:26:37.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.770+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9c0fc920 con 0x7f7a9c101b50 2024-06-13T01:26:37.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.770+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7a88012070 con 0x7f7a9c101b50 2024-06-13T01:26:37.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.771+0000 7f7a997fa640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7a8c065430 0x7f7a8c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:37.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.771+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7a880820d0 con 0x7f7a9c101b50 2024-06-13T01:26:37.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.771+0000 7f7aa0d53640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7a8c065430 0x7f7a8c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:37.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.771+0000 7f7aa0d53640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7a8c065430 0x7f7a8c0678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a900047f0 tx=0x7f7a90009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:37.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.775+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7a88050580 con 0x7f7a9c101b50 2024-06-13T01:26:37.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.973+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7a9c0fa9a0 con 0x7f7a9c101b50 2024-06-13T01:26:37.973 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.974+0000 7f7a997fa640 1 -- 172.21.15.132:0/2908268518 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7a88053ba0 con 0x7f7a9c101b50 2024-06-13T01:26:37.973 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:37.974 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:37.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.975+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7a8c065430 msgr2=0x7f7a8c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:37.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.975+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7a8c065430 0x7f7a8c0678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a900047f0 tx=0x7f7a90009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.975+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 msgr2=0x7f7a9c112500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.975+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 0x7f7a9c112500 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800b790 tx=0x7f7a880091d0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 shutdown_connections 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7a8c065430 0x7f7a8c0678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a9c101b50 0x7f7a9c112500 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 --2- 172.21.15.132:0/2908268518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a9c1011b0 0x7f7a9c111fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 >> 172.21.15.132:0/2908268518 conn(0x7f7a9c0f6d60 msgr2=0x7f7a9c0f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 shutdown_connections 2024-06-13T01:26:37.976 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:37.976+0000 7f7aa2fb7640 1 -- 172.21.15.132:0/2908268518 wait complete. 2024-06-13T01:26:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:38 smithi132 ceph-mon[29330]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:39.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:38 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2908268518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:39.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:38 smithi155 ceph-mon[35815]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:39.379 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:38 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2908268518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:40.378 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:39 smithi155 ceph-mon[35815]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:40.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:39 smithi132 ceph-mon[29330]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:41.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:41.961 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:42.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:42 smithi132 ceph-mon[29330]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:42.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:42 smithi155 ceph-mon[35815]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:43.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- 172.21.15.132:0/2846035686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 msgr2=0x7fdc800faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:43.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 --2- 172.21.15.132:0/2846035686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc800faec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009c50 tx=0x7fdc74004d10 comp rx=0 tx=0).stop 2024-06-13T01:26:43.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- 172.21.15.132:0/2846035686 shutdown_connections 2024-06-13T01:26:43.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 --2- 172.21.15.132:0/2846035686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc800faec0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 --2- 172.21.15.132:0/2846035686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 0x7fdc801027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- 172.21.15.132:0/2846035686 >> 172.21.15.132:0/2846035686 conn(0x7fdc80070ab0 msgr2=0x7fdc80070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- 172.21.15.132:0/2846035686 shutdown_connections 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- 172.21.15.132:0/2846035686 wait complete. 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 Processor -- start 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- start start 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc801130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 0x7fdc80113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc80113c00 con 0x7fdc800faa50 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.559+0000 7fdc8806e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8010d040 con 0x7fdc801023e0 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc801130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc801130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33708/0 (socket says 172.21.15.132:33708) 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 -- 172.21.15.132:0/3170053368 learned_addr learned my addr 172.21.15.132:0/3170053368 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:43.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85609640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 0x7fdc80113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 -- 172.21.15.132:0/3170053368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 msgr2=0x7fdc80113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 0x7fdc80113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 -- 172.21.15.132:0/3170053368 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc740098b0 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85609640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 0x7fdc80113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc85e0a640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc801130c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800c810 tx=0x7fdc780043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc7800a820 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc7800ae50 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc780115c0 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8010d320 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.560+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8010d7f0 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.561+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc80065550 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.562+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdc7800a980 con 0x7fdc800faa50 2024-06-13T01:26:43.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.562+0000 7fdc7e7fc640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fdc68065430 0x7fdc680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:43.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.562+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc78082a60 con 0x7fdc800faa50 2024-06-13T01:26:43.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.562+0000 7fdc85609640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fdc68065430 0x7fdc680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:43.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.562+0000 7fdc85609640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fdc68065430 0x7fdc680678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdc74004f50 tx=0x7fdc74004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:43.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.566+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdc78050f10 con 0x7fdc800faa50 2024-06-13T01:26:43.764 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.764+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc8004b480 con 0x7fdc800faa50 2024-06-13T01:26:43.765 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.765+0000 7fdc7e7fc640 1 -- 172.21.15.132:0/3170053368 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdc78054530 con 0x7fdc800faa50 2024-06-13T01:26:43.765 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:43.765 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:43.766 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fdc68065430 msgr2=0x7fdc680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fdc68065430 0x7fdc680678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdc74004f50 tx=0x7fdc74004d10 comp rx=0 tx=0).stop 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 msgr2=0x7fdc801130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc801130c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800c810 tx=0x7fdc780043c0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 shutdown_connections 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fdc68065430 0x7fdc680678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc801023e0 0x7fdc80113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 --2- 172.21.15.132:0/3170053368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc800faa50 0x7fdc801130c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 >> 172.21.15.132:0/3170053368 conn(0x7fdc80070ab0 msgr2=0x7fdc800f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 shutdown_connections 2024-06-13T01:26:43.767 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:43.767+0000 7fdc8806e640 1 -- 172.21.15.132:0/3170053368 wait complete. 2024-06-13T01:26:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:44 smithi132 ceph-mon[29330]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:44.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:44 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3170053368' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:44 smithi155 ceph-mon[35815]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:44 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3170053368' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:46.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:46 smithi132 ceph-mon[29330]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:46.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:46 smithi155 ceph-mon[35815]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:47.439 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:47.703 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:48.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:48 smithi132 ceph-mon[29330]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:48.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:48 smithi155 ceph-mon[35815]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:49.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/3536490270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 msgr2=0x7f5e9c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:49.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/3536490270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c102690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0048d0 tx=0x7f5e8c009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/3536490270 shutdown_connections 2024-06-13T01:26:49.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/3536490270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e9c102c60 0x7f5e9c0fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/3536490270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c102690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/3536490270 >> 172.21.15.132:0/3536490270 conn(0x7f5e9c0f7e50 msgr2=0x7f5e9c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/3536490270 shutdown_connections 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/3536490270 wait complete. 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 Processor -- start 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.320+0000 7f5ea0e6a640 1 -- start start 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5ea0e6a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c113310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5ea0e6a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e9c102c60 0x7f5e9c113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5ea0e6a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c113e50 con 0x7f5e9c1022c0 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5ea0e6a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c10d290 con 0x7f5e9c102c60 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c113310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c113310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33724/0 (socket says 172.21.15.132:33724) 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 -- 172.21.15.132:0/625793635 learned_addr learned my addr 172.21.15.132:0/625793635 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:49.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e93fff640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e9c102c60 0x7f5e9c113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 -- 172.21.15.132:0/625793635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e9c102c60 msgr2=0x7f5e9c113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e9c102c60 0x7f5e9c113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 -- 172.21.15.132:0/625793635 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c005040 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e9a59c640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c113310 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c003080 tx=0x7f5e8c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e8c038070 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e8c032970 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c10d510 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e8c03d960 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.321+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c10da50 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.322+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c0fda30 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.323+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5e8c030020 con 0x7f5e9c1022c0 2024-06-13T01:26:49.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.323+0000 7f5e92ffd640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5e88065430 0x7f5e880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:49.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.324+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5e8c0a2540 con 0x7f5e9c1022c0 2024-06-13T01:26:49.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.324+0000 7f5e93fff640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5e88065430 0x7f5e880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:49.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.324+0000 7f5e93fff640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5e88065430 0x7f5e880678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c10ebd0 tx=0x7f5e8400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:49.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.331+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5e8c0709f0 con 0x7f5e9c1022c0 2024-06-13T01:26:49.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.545+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5e9c04b480 con 0x7f5e9c1022c0 2024-06-13T01:26:49.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.546+0000 7f5e92ffd640 1 -- 172.21.15.132:0/625793635 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5e8c074010 con 0x7f5e9c1022c0 2024-06-13T01:26:49.546 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:49.546 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:49.547 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5e88065430 msgr2=0x7f5e880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5e88065430 0x7f5e880678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c10ebd0 tx=0x7f5e8400b040 comp rx=0 tx=0).stop 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 msgr2=0x7f5e9c113310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c113310 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c003080 tx=0x7f5e8c009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 shutdown_connections 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f5e88065430 0x7f5e880678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e9c102c60 0x7f5e9c113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 --2- 172.21.15.132:0/625793635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e9c1022c0 0x7f5e9c113310 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 >> 172.21.15.132:0/625793635 conn(0x7f5e9c0f7e50 msgr2=0x7f5e9c0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 shutdown_connections 2024-06-13T01:26:49.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:49.548+0000 7f5ea0e6a640 1 -- 172.21.15.132:0/625793635 wait complete. 2024-06-13T01:26:49.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:26:49.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:49.919 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:26:49.919 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:26:50.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:50 smithi132 ceph-mon[29330]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:50.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:26:50.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:26:50.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:26:50.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:50 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/625793635' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:50 smithi155 ceph-mon[35815]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:26:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:26:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:26:50.879 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:50 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/625793635' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:52 smithi155 ceph-mon[35815]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:52.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:52 smithi132 ceph-mon[29330]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:53.305 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:53.568 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:26:54.723 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:54 smithi132 ceph-mon[29330]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:54.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:54 smithi155 ceph-mon[35815]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 -- 172.21.15.132:0/180729099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 msgr2=0x7f42241026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 --2- 172.21.15.132:0/180729099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f42241026a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f42140048d0 tx=0x7f4214009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 -- 172.21.15.132:0/180729099 shutdown_connections 2024-06-13T01:26:55.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 --2- 172.21.15.132:0/180729099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4224102c70 0x7f42240fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 --2- 172.21.15.132:0/180729099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f42241026a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 -- 172.21.15.132:0/180729099 >> 172.21.15.132:0/180729099 conn(0x7f42240f7e80 msgr2=0x7f42240fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 -- 172.21.15.132:0/180729099 shutdown_connections 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.081+0000 7f422a516640 1 -- 172.21.15.132:0/180729099 wait complete. 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f422a516640 1 Processor -- start 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f422a516640 1 -- start start 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f422a516640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f422406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f422a516640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4224102c70 0x7f422406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f422a516640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240701f0 con 0x7f42241022d0 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f422a516640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4224070360 con 0x7f4224102c70 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f422406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:55.098 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f422406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33984/0 (socket says 172.21.15.132:33984) 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 -- 172.21.15.132:0/2747630517 learned_addr learned my addr 172.21.15.132:0/2747630517 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f421bfff640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4224102c70 0x7f422406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 -- 172.21.15.132:0/2747630517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4224102c70 msgr2=0x7f422406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4224102c70 0x7f422406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 -- 172.21.15.132:0/2747630517 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4214005040 con 0x7f42241022d0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.082+0000 7f4223fff640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f422406f770 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f42140043d0 tx=0x7f4214009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.083+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4214038070 con 0x7f42241022d0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.083+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4214032970 con 0x7f42241022d0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.083+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f421403da60 con 0x7f42241022d0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.083+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4224070590 con 0x7f42241022d0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.083+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4224119680 con 0x7f42241022d0 2024-06-13T01:26:55.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.084+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42240fda40 con 0x7f42241022d0 2024-06-13T01:26:55.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.084+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4214030020 con 0x7f42241022d0 2024-06-13T01:26:55.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.084+0000 7f42217fa640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4210065430 0x7f42100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:26:55.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.085+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f42140a25e0 con 0x7f42241022d0 2024-06-13T01:26:55.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.085+0000 7f421bfff640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4210065430 0x7f42100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:26:55.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.085+0000 7f421bfff640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4210065430 0x7f42100678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f42240710b0 tx=0x7f420c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:26:55.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.089+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4214070a90 con 0x7f42241022d0 2024-06-13T01:26:55.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.286+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f42240fbac0 con 0x7f42241022d0 2024-06-13T01:26:55.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.286+0000 7f42217fa640 1 -- 172.21.15.132:0/2747630517 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f42140740b0 con 0x7f42241022d0 2024-06-13T01:26:55.294 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:26:55.294 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:26:55.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4210065430 msgr2=0x7f42100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:55.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4210065430 0x7f42100678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f42240710b0 tx=0x7f420c00b040 comp rx=0 tx=0).stop 2024-06-13T01:26:55.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 msgr2=0x7f422406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:26:55.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f422406f770 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f42140043d0 tx=0x7f4214009da0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 shutdown_connections 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4210065430 0x7f42100678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4224102c70 0x7f422406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 --2- 172.21.15.132:0/2747630517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42241022d0 0x7f422406f770 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 >> 172.21.15.132:0/2747630517 conn(0x7f42240f7e80 msgr2=0x7f42240f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 shutdown_connections 2024-06-13T01:26:55.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:26:55.288+0000 7f422a516640 1 -- 172.21.15.132:0/2747630517 wait complete. 2024-06-13T01:26:55.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:55 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2747630517' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:55.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:55 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2747630517' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:26:56.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:56 smithi155 ceph-mon[35815]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:56.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:56 smithi132 ceph-mon[29330]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:58.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:26:58 smithi132 ceph-mon[29330]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:26:58 smithi155 ceph-mon[35815]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:26:58.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:26:59.193 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:00.603 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:00 smithi132 ceph-mon[29330]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:00.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 -- 172.21.15.132:0/2519936860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 msgr2=0x7fbb901026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:00.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/2519936860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb901026c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbb840048d0 tx=0x7fbb84009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:00.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 -- 172.21.15.132:0/2519936860 shutdown_connections 2024-06-13T01:27:00.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/2519936860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb90102c90 0x7fbb900fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:00.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/2519936860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb901026c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 -- 172.21.15.132:0/2519936860 >> 172.21.15.132:0/2519936860 conn(0x7fbb90070a40 msgr2=0x7fbb90070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 -- 172.21.15.132:0/2519936860 shutdown_connections 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.795+0000 7fbb94c0e640 1 -- 172.21.15.132:0/2519936860 wait complete. 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb94c0e640 1 Processor -- start 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb94c0e640 1 -- start start 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb94c0e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb90113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb94c0e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb90102c90 0x7fbb901136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb94c0e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90113d40 con 0x7fbb901022f0 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb94c0e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9010d0f0 con 0x7fbb90102c90 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb90113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb90113170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34004/0 (socket says 172.21.15.132:34004) 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 -- 172.21.15.132:0/1550919321 learned_addr learned my addr 172.21.15.132:0/1550919321 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:00.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 -- 172.21.15.132:0/1550919321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb90102c90 msgr2=0x7fbb901136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb90102c90 0x7fbb901136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 -- 172.21.15.132:0/1550919321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb84005040 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.797+0000 7fbb8e59c640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb90113170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb84008fc0 tx=0x7fbb84009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.798+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb84038070 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.798+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb84032b80 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.798+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9010d370 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.798+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb8403dc20 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.798+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9010d830 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.799+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb90065510 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.799+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fbb84030020 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.799+0000 7fbb7effd640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fbb78065430 0x7fbb780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.799+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbb840a2ee0 con 0x7fbb901022f0 2024-06-13T01:27:00.822 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.799+0000 7fbb8dd9b640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fbb78065430 0x7fbb780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:00.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.800+0000 7fbb8dd9b640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fbb78065430 0x7fbb780678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbb9010ea30 tx=0x7fbb80006d20 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:00.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:00.804+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbb84071390 con 0x7fbb901022f0 2024-06-13T01:27:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:00 smithi155 ceph-mon[35815]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:01.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.002+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbb90074960 con 0x7fbb901022f0 2024-06-13T01:27:01.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.003+0000 7fbb7effd640 1 -- 172.21.15.132:0/1550919321 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbb90074960 con 0x7fbb901022f0 2024-06-13T01:27:01.003 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:01.003 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fbb78065430 msgr2=0x7fbb780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fbb78065430 0x7fbb780678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbb9010ea30 tx=0x7fbb80006d20 comp rx=0 tx=0).stop 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 msgr2=0x7fbb90113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb90113170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb84008fc0 tx=0x7fbb84009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 shutdown_connections 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fbb78065430 0x7fbb780678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb90102c90 0x7fbb901136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 --2- 172.21.15.132:0/1550919321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb901022f0 0x7fbb90113170 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 >> 172.21.15.132:0/1550919321 conn(0x7fbb90070a40 msgr2=0x7fbb901050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 shutdown_connections 2024-06-13T01:27:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:01.005+0000 7fbb94c0e640 1 -- 172.21.15.132:0/1550919321 wait complete. 2024-06-13T01:27:01.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:01 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1550919321' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:01.779 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:01 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1550919321' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:02 smithi155 ceph-mon[35815]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:02.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:02 smithi132 ceph-mon[29330]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:04.750 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:04.762 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:04 smithi132 ceph-mon[29330]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:04.763 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:04.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:04 smithi155 ceph-mon[35815]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:04.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:05.028 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:06.694 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:06 smithi132 ceph-mon[29330]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:06.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 -- 172.21.15.132:0/619568431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540faa50 msgr2=0x7fc7540faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:06.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/619568431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540faa50 0x7fc7540faec0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc7480048d0 tx=0x7fc748009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 -- 172.21.15.132:0/619568431 shutdown_connections 2024-06-13T01:27:06.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/619568431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7540faa50 0x7fc7540faec0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/619568431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7541023e0 0x7fc7541027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 -- 172.21.15.132:0/619568431 >> 172.21.15.132:0/619568431 conn(0x7fc754070ab0 msgr2=0x7fc754070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 -- 172.21.15.132:0/619568431 shutdown_connections 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 -- 172.21.15.132:0/619568431 wait complete. 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 Processor -- start 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.696+0000 7fc75bbc8640 1 -- start start 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc75bbc8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540faa50 0x7fc7540fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc75bbc8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 0x7fc7540fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc75bbc8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540fc6b0 con 0x7fc7541023e0 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc75bbc8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540fc820 con 0x7fc7540faa50 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 0x7fc7540fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:06.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 0x7fc7540fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46754/0 (socket says 172.21.15.132:46754) 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 -- 172.21.15.132:0/749797579 learned_addr learned my addr 172.21.15.132:0/749797579 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759964640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540faa50 0x7fc7540fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 -- 172.21.15.132:0/749797579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540faa50 msgr2=0x7fc7540fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540faa50 0x7fc7540fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 -- 172.21.15.132:0/749797579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748005040 con 0x7fc7541023e0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759964640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540faa50 0x7fc7540fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc759163640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 0x7fc7540fc170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc7480043d0 tx=0x7fc748009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc748038070 con 0x7fc7541023e0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7540fca50 con 0x7fc7541023e0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc748032970 con 0x7fc7541023e0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.697+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc74803da60 con 0x7fc7541023e0 2024-06-13T01:27:06.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.698+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7541196c0 con 0x7fc7541023e0 2024-06-13T01:27:06.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.698+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc754065550 con 0x7fc7541023e0 2024-06-13T01:27:06.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.699+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc748030020 con 0x7fc7541023e0 2024-06-13T01:27:06.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.699+0000 7fc7527fc640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc73c065430 0x7fc73c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:06.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.699+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc7480a26c0 con 0x7fc7541023e0 2024-06-13T01:27:06.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.699+0000 7fc759964640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc73c065430 0x7fc73c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:06.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.700+0000 7fc759964640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc73c065430 0x7fc73c0678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc744007fc0 tx=0x7fc744004d00 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:06.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.703+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc748070b70 con 0x7fc7541023e0 2024-06-13T01:27:06.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:06 smithi155 ceph-mon[35815]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:06.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.902+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7541095c0 con 0x7fc7541023e0 2024-06-13T01:27:06.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.902+0000 7fc7527fc640 1 -- 172.21.15.132:0/749797579 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc748074190 con 0x7fc7541023e0 2024-06-13T01:27:06.902 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:06.902 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc73c065430 msgr2=0x7fc73c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc73c065430 0x7fc73c0678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc744007fc0 tx=0x7fc744004d00 comp rx=0 tx=0).stop 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 msgr2=0x7fc7540fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 0x7fc7540fc170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc7480043d0 tx=0x7fc748009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 shutdown_connections 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc73c065430 0x7fc73c0678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7541023e0 0x7fc7540fc170 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 --2- 172.21.15.132:0/749797579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7540faa50 0x7fc7540fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:06.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 >> 172.21.15.132:0/749797579 conn(0x7fc754070ab0 msgr2=0x7fc7540f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:06.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 shutdown_connections 2024-06-13T01:27:06.905 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:06.904+0000 7fc75bbc8640 1 -- 172.21.15.132:0/749797579 wait complete. 2024-06-13T01:27:07.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:07 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/749797579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:07.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:07 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/749797579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:08.713 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:08 smithi132 ceph-mon[29330]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:08.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:08 smithi155 ceph-mon[35815]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:10.611 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:10.877 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:10.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:10 smithi155 ceph-mon[35815]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:10.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:10 smithi132 ceph-mon[29330]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:12.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 -- 172.21.15.132:0/3891802643 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 msgr2=0x7f49900fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 --2- 172.21.15.132:0/3891802643 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f49900fc2d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f497c0048d0 tx=0x7f497c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 -- 172.21.15.132:0/3891802643 shutdown_connections 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 --2- 172.21.15.132:0/3891802643 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f49900fc2d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 --2- 172.21.15.132:0/3891802643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49901022a0 0x7f4990102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 -- 172.21.15.132:0/3891802643 >> 172.21.15.132:0/3891802643 conn(0x7f49900f7e70 msgr2=0x7f49900fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 -- 172.21.15.132:0/3891802643 shutdown_connections 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.503+0000 7f4997224640 1 -- 172.21.15.132:0/3891802643 wait complete. 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4997224640 1 Processor -- start 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4997224640 1 -- start start 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4997224640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49901022a0 0x7f49901130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4997224640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f4990113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:12.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4997224640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4990113c90 con 0x7f4990102c40 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4997224640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010d040 con 0x7f49901022a0 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f498ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f4990113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f498ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f4990113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52394/0 (socket says 172.21.15.132:52394) 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f498ffff640 1 -- 172.21.15.132:0/2015744757 learned_addr learned my addr 172.21.15.132:0/2015744757 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.504+0000 7f4994fc0640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49901022a0 0x7f49901130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.505+0000 7f498ffff640 1 -- 172.21.15.132:0/2015744757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49901022a0 msgr2=0x7f49901130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.505+0000 7f498ffff640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49901022a0 0x7f49901130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.505+0000 7f498ffff640 1 -- 172.21.15.132:0/2015744757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c005040 con 0x7f4990102c40 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.505+0000 7f498ffff640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f4990113600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f497c003080 tx=0x7f497c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.506+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c038070 con 0x7f4990102c40 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.506+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f497c008ee0 con 0x7f4990102c40 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.506+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499010d2c0 con 0x7f4990102c40 2024-06-13T01:27:12.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.506+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c03d870 con 0x7f4990102c40 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.506+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499010d780 con 0x7f4990102c40 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.507+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49900fda10 con 0x7f4990102c40 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.508+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f497c030020 con 0x7f4990102c40 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.508+0000 7f498d7fa640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4980065430 0x7f49800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.509+0000 7f4994fc0640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4980065430 0x7f49800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.509+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f497c0a20c0 con 0x7f4990102c40 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.509+0000 7f4994fc0640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4980065430 0x7f49800678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4984009ac0 tx=0x7f4984009120 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:12.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.513+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f497c070690 con 0x7f4990102c40 2024-06-13T01:27:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.711+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f49900fba90 con 0x7f4990102c40 2024-06-13T01:27:12.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:12 smithi132 ceph-mon[29330]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:12.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.711+0000 7f498d7fa640 1 -- 172.21.15.132:0/2015744757 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f497c073cb0 con 0x7f4990102c40 2024-06-13T01:27:12.711 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:12.712 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4980065430 msgr2=0x7f49800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4980065430 0x7f49800678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4984009ac0 tx=0x7f4984009120 comp rx=0 tx=0).stop 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 msgr2=0x7f4990113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f4990113600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f497c003080 tx=0x7f497c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 shutdown_connections 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4980065430 0x7f49800678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.713 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4990102c40 0x7f4990113600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 --2- 172.21.15.132:0/2015744757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49901022a0 0x7f49901130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 >> 172.21.15.132:0/2015744757 conn(0x7f49900f7e70 msgr2=0x7f49900f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 shutdown_connections 2024-06-13T01:27:12.714 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:12.713+0000 7f4997224640 1 -- 172.21.15.132:0/2015744757 wait complete. 2024-06-13T01:27:12.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:12 smithi155 ceph-mon[35815]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:13.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:13 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2015744757' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:13.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:13 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2015744757' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:14.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:14 smithi155 ceph-mon[35815]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:14.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:14 smithi132 ceph-mon[29330]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:16.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:16.584 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:16.705 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:16 smithi132 ceph-mon[29330]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:16.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:16 smithi155 ceph-mon[35815]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:18.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3044764251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fd100 msgr2=0x7f9e300fd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:18.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3044764251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fd100 0x7f9e300fd4d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0048d0 tx=0x7f9e2c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3044764251 shutdown_connections 2024-06-13T01:27:18.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3044764251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e300fa9e0 0x7f9e300fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3044764251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fd100 0x7f9e300fd4d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3044764251 >> 172.21.15.132:0/3044764251 conn(0x7f9e30070a40 msgr2=0x7f9e30070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3044764251 shutdown_connections 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3044764251 wait complete. 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 Processor -- start 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- start start 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 0x7f9e30115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e300fd100 0x7f9e301157b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e30115db0 con 0x7f9e300fa9e0 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e37ef0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010f1f0 con 0x7f9e300fd100 2024-06-13T01:27:18.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.226+0000 7f9e35c8c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 0x7f9e30115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e35c8c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 0x7f9e30115270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52414/0 (socket says 172.21.15.132:52414) 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e35c8c640 1 -- 172.21.15.132:0/3378239765 learned_addr learned my addr 172.21.15.132:0/3378239765 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e3548b640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e300fd100 0x7f9e301157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e35c8c640 1 -- 172.21.15.132:0/3378239765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e300fd100 msgr2=0x7f9e301157b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e35c8c640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e300fd100 0x7f9e301157b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e35c8c640 1 -- 172.21.15.132:0/3378239765 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c005040 con 0x7f9e300fa9e0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e35c8c640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 0x7f9e30115270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c003080 tx=0x7f9e2c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2c038070 con 0x7f9e300fa9e0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3010f470 con 0x7f9e300fa9e0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e2c008ee0 con 0x7f9e300fa9e0 2024-06-13T01:27:18.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2c03d870 con 0x7f9e300fa9e0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.227+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3010f930 con 0x7f9e300fa9e0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.228+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e300fb9f0 con 0x7f9e300fa9e0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.228+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9e2c030020 con 0x7f9e300fa9e0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.229+0000 7f9e267fc640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9e1c065430 0x7f9e1c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.229+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9e2c0a2250 con 0x7f9e300fa9e0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.229+0000 7f9e3548b640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9e1c065430 0x7f9e1c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.229+0000 7f9e3548b640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9e1c065430 0x7f9e1c0678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e20007fd0 tx=0x7f9e20009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:18.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.233+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9e2c070780 con 0x7f9e300fa9e0 2024-06-13T01:27:18.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.432+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9e3010b620 con 0x7f9e300fa9e0 2024-06-13T01:27:18.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.432+0000 7f9e267fc640 1 -- 172.21.15.132:0/3378239765 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9e2c073da0 con 0x7f9e300fa9e0 2024-06-13T01:27:18.434 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:18.434 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9e1c065430 msgr2=0x7f9e1c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9e1c065430 0x7f9e1c0678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e20007fd0 tx=0x7f9e20009040 comp rx=0 tx=0).stop 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 msgr2=0x7f9e30115270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 0x7f9e30115270 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c003080 tx=0x7f9e2c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 shutdown_connections 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f9e1c065430 0x7f9e1c0678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e300fd100 0x7f9e301157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 --2- 172.21.15.132:0/3378239765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e300fa9e0 0x7f9e30115270 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:18.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.434+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 >> 172.21.15.132:0/3378239765 conn(0x7f9e30070a40 msgr2=0x7f9e300f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:18.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.435+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 shutdown_connections 2024-06-13T01:27:18.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:18.435+0000 7f9e37ef0640 1 -- 172.21.15.132:0/3378239765 wait complete. 2024-06-13T01:27:18.786 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:18 smithi132 ceph-mon[29330]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:18.786 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:18 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3378239765' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:18.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:18 smithi155 ceph-mon[35815]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:18.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:18 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3378239765' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:19.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:19.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:20.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:20 smithi155 ceph-mon[35815]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:20.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:20 smithi132 ceph-mon[29330]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:22.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:22.428 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:22.841 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:22 smithi132 ceph-mon[29330]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:22.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:22 smithi155 ceph-mon[35815]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 -- 172.21.15.132:0/291863055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1480fdea0 msgr2=0x7fa1480fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 --2- 172.21.15.132:0/291863055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1480fdea0 0x7fa1480fe310 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0048d0 tx=0x7fa13c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 -- 172.21.15.132:0/291863055 shutdown_connections 2024-06-13T01:27:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 --2- 172.21.15.132:0/291863055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1480fdea0 0x7fa1480fe310 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 --2- 172.21.15.132:0/291863055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa148065010 0x7fa1480653e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 -- 172.21.15.132:0/291863055 >> 172.21.15.132:0/291863055 conn(0x7fa148072110 msgr2=0x7fa148072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 -- 172.21.15.132:0/291863055 shutdown_connections 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.041+0000 7fa14de90640 1 -- 172.21.15.132:0/291863055 wait complete. 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa14de90640 1 Processor -- start 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa14de90640 1 -- start start 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa14de90640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 0x7fa1481130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa14de90640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1480fdea0 0x7fa1481135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa14de90640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148113c70 con 0x7fa148065010 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa14de90640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14810d060 con 0x7fa1480fdea0 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa1477fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 0x7fa1481130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa1477fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 0x7fa1481130a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44284/0 (socket says 172.21.15.132:44284) 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.042+0000 7fa1477fe640 1 -- 172.21.15.132:0/120870430 learned_addr learned my addr 172.21.15.132:0/120870430 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:24.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa146ffd640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1480fdea0 0x7fa1481135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa1477fe640 1 -- 172.21.15.132:0/120870430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1480fdea0 msgr2=0x7fa1481135e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa1477fe640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1480fdea0 0x7fa1481135e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa1477fe640 1 -- 172.21.15.132:0/120870430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c005040 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa1477fe640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 0x7fa1481130a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa13800b790 tx=0x7fa1380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa13800c700 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa14de90640 1 -- 172.21.15.132:0/120870430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14810d340 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa13800cd30 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa138013370 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.043+0000 7fa14de90640 1 -- 172.21.15.132:0/120870430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14810d830 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.044+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa128005210 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.045+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa138012070 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.045+0000 7fa12ffff640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa130065430 0x7fa1300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.045+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa138081dd0 con 0x7fa148065010 2024-06-13T01:27:24.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.045+0000 7fa146ffd640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa130065430 0x7fa1300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.046+0000 7fa146ffd640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa130065430 0x7fa1300678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009f80 tx=0x7fa13c035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:24.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.049+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa138050280 con 0x7fa148065010 2024-06-13T01:27:24.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.248+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa128005a40 con 0x7fa148065010 2024-06-13T01:27:24.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.249+0000 7fa12ffff640 1 -- 172.21.15.132:0/120870430 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa1380538a0 con 0x7fa148065010 2024-06-13T01:27:24.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:24.251 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa130065430 msgr2=0x7fa1300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa130065430 0x7fa1300678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009f80 tx=0x7fa13c035040 comp rx=0 tx=0).stop 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 msgr2=0x7fa1481130a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 0x7fa1481130a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa13800b790 tx=0x7fa1380091d0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 shutdown_connections 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fa130065430 0x7fa1300678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1480fdea0 0x7fa1481135e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 --2- 172.21.15.132:0/120870430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa148065010 0x7fa1481130a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:24.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 >> 172.21.15.132:0/120870430 conn(0x7fa148072110 msgr2=0x7fa1480f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:24.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 shutdown_connections 2024-06-13T01:27:24.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:24.251+0000 7fa12dffb640 1 -- 172.21.15.132:0/120870430 wait complete. 2024-06-13T01:27:24.843 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:24 smithi132 ceph-mon[29330]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:24.844 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:24 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/120870430' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:24.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:24 smithi155 ceph-mon[35815]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:24.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:24 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/120870430' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:26.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:26 smithi155 ceph-mon[35815]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:26.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:26 smithi132 ceph-mon[29330]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:27.869 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:28.132 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:28.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:28 smithi155 ceph-mon[35815]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:28.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:28 smithi132 ceph-mon[29330]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:29.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 -- 172.21.15.132:0/3108537310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 msgr2=0x7f11f00fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:29.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 --2- 172.21.15.132:0/3108537310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fb1f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009c50 tx=0x7f11e4004d10 comp rx=0 tx=0).stop 2024-06-13T01:27:29.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 -- 172.21.15.132:0/3108537310 shutdown_connections 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 --2- 172.21.15.132:0/3108537310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fb1f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 --2- 172.21.15.132:0/3108537310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f01011c0 0x7f11f0101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 -- 172.21.15.132:0/3108537310 >> 172.21.15.132:0/3108537310 conn(0x7f11f00f6db0 msgr2=0x7f11f00f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 -- 172.21.15.132:0/3108537310 shutdown_connections 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.689+0000 7f11f8628640 1 -- 172.21.15.132:0/3108537310 wait complete. 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f8628640 1 Processor -- start 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f8628640 1 -- start start 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f8628640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f01011c0 0x7f11f00fce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f8628640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f8628640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00fd8e0 con 0x7f11f0101b60 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f8628640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00fda50 con 0x7f11f01011c0 2024-06-13T01:27:29.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f5bc3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:29.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f5bc3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44298/0 (socket says 172.21.15.132:44298) 2024-06-13T01:27:29.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.690+0000 7f11f5bc3640 1 -- 172.21.15.132:0/2885685561 learned_addr learned my addr 172.21.15.132:0/2885685561 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:29.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f63c4640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f01011c0 0x7f11f00fce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:29.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f5bc3640 1 -- 172.21.15.132:0/2885685561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f01011c0 msgr2=0x7f11f00fce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:29.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f5bc3640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f01011c0 0x7f11f00fce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:29.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f5bc3640 1 -- 172.21.15.132:0/2885685561 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e40098b0 con 0x7f11f0101b60 2024-06-13T01:27:29.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f5bc3640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fd3a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f11e4004f80 tx=0x7f11e4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e4008de0 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11e4032c00 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f00fdc80 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11e403dbc0 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.691+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f0119670 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.692+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f00fc930 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.693+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f11e402f030 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.693+0000 7f11eeffd640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f11d8065430 0x7f11d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.693+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f11e40a3ae0 con 0x7f11f0101b60 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.693+0000 7f11f63c4640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f11d8065430 0x7f11d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:29.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.693+0000 7f11f63c4640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f11d8065430 0x7f11d80678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11e8007fd0 tx=0x7f11e8009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:29.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.697+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f11e4072010 con 0x7f11f0101b60 2024-06-13T01:27:29.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.893+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f11f00fa9b0 con 0x7f11f0101b60 2024-06-13T01:27:29.893 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.894+0000 7f11eeffd640 1 -- 172.21.15.132:0/2885685561 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f11e4075630 con 0x7f11f0101b60 2024-06-13T01:27:29.894 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:29.894 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:29.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f11d8065430 msgr2=0x7f11d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:29.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f11d8065430 0x7f11d80678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f11e8007fd0 tx=0x7f11e8009040 comp rx=0 tx=0).stop 2024-06-13T01:27:29.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 msgr2=0x7f11f00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fd3a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f11e4004f80 tx=0x7f11e4004d10 comp rx=0 tx=0).stop 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 shutdown_connections 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f11d8065430 0x7f11d80678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11f0101b60 0x7f11f00fd3a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 --2- 172.21.15.132:0/2885685561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f01011c0 0x7f11f00fce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.895+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 >> 172.21.15.132:0/2885685561 conn(0x7f11f00f6db0 msgr2=0x7f11f00f8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.896+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 shutdown_connections 2024-06-13T01:27:29.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:29.896+0000 7f11f8628640 1 -- 172.21.15.132:0/2885685561 wait complete. 2024-06-13T01:27:30.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:30 smithi132 ceph-mon[29330]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:30.600 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:30 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2885685561' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:30.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:30 smithi155 ceph-mon[35815]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:30.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:30 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2885685561' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:32.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:32 smithi155 ceph-mon[35815]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:32.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:32 smithi132 ceph-mon[29330]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:33.626 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:33.899 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:34.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:34 smithi132 ceph-mon[29330]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:34.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:34.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:34 smithi155 ceph-mon[35815]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:34.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.544+0000 7f7f4243d640 1 -- 172.21.15.132:0/4150020726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 msgr2=0x7f7f3c102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.544+0000 7f7f4243d640 1 --2- 172.21.15.132:0/4150020726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c102670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0048d0 tx=0x7f7f2c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 -- 172.21.15.132:0/4150020726 shutdown_connections 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 --2- 172.21.15.132:0/4150020726 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f3c102c40 0x7f7f3c0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 --2- 172.21.15.132:0/4150020726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c102670 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 -- 172.21.15.132:0/4150020726 >> 172.21.15.132:0/4150020726 conn(0x7f7f3c0f7e70 msgr2=0x7f7f3c0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:35.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 -- 172.21.15.132:0/4150020726 shutdown_connections 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 -- 172.21.15.132:0/4150020726 wait complete. 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 Processor -- start 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.545+0000 7f7f4243d640 1 -- start start 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f4243d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c1152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f4243d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f3c102c40 0x7f7f3c1157e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f4243d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c115e70 con 0x7f7f3c1022a0 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f4243d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c10f220 con 0x7f7f3c102c40 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c1152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c1152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34496/0 (socket says 172.21.15.132:34496) 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 -- 172.21.15.132:0/968978717 learned_addr learned my addr 172.21.15.132:0/968978717 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3b7fe640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f3c102c40 0x7f7f3c1157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:35.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 -- 172.21.15.132:0/968978717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f3c102c40 msgr2=0x7f7f3c1157e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f3c102c40 0x7f7f3c1157e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 -- 172.21.15.132:0/968978717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c005040 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.546+0000 7f7f3bfff640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c1152a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c003080 tx=0x7f7f2c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.547+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2c038070 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.547+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f2c008ee0 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.547+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c10f4a0 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.547+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2c03d870 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.547+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c10f960 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.548+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c0fda10 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.549+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7f2c030020 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.549+0000 7f7f38ff9640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7f24065430 0x7f7f240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.549+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7f2c0a2170 con 0x7f7f3c1022a0 2024-06-13T01:27:35.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.549+0000 7f7f3b7fe640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7f24065430 0x7f7f240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:35.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.549+0000 7f7f3b7fe640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7f24065430 0x7f7f240678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c110b60 tx=0x7f7f30008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:35.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.553+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7f2c070740 con 0x7f7f3c1022a0 2024-06-13T01:27:35.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.750+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7f3c04b4b0 con 0x7f7f3c1022a0 2024-06-13T01:27:35.750 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.750+0000 7f7f38ff9640 1 -- 172.21.15.132:0/968978717 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7f2c073d60 con 0x7f7f3c1022a0 2024-06-13T01:27:35.750 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:35.751 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7f24065430 msgr2=0x7f7f240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7f24065430 0x7f7f240678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c110b60 tx=0x7f7f30008040 comp rx=0 tx=0).stop 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 msgr2=0x7f7f3c1152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c1152a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c003080 tx=0x7f7f2c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 shutdown_connections 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7f24065430 0x7f7f240678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f3c102c40 0x7f7f3c1157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 --2- 172.21.15.132:0/968978717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f3c1022a0 0x7f7f3c1152a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:35.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 >> 172.21.15.132:0/968978717 conn(0x7f7f3c0f7e70 msgr2=0x7f7f3c0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:35.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 shutdown_connections 2024-06-13T01:27:35.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:35.753+0000 7f7f4243d640 1 -- 172.21.15.132:0/968978717 wait complete. 2024-06-13T01:27:36.597 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:36 smithi132 ceph-mon[29330]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:36.597 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:36 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/968978717' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:36.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:36 smithi155 ceph-mon[35815]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:36.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:36 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/968978717' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:38.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:38 smithi132 ceph-mon[29330]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:38.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:38 smithi155 ceph-mon[35815]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:39.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:39.607 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:40.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:40 smithi155 ceph-mon[35815]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:40.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:40 smithi132 ceph-mon[29330]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.219+0000 7f45f3af3640 1 -- 172.21.15.132:0/3165498925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec102c90 msgr2=0x7f45ec0fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.219+0000 7f45f3af3640 1 --2- 172.21.15.132:0/3165498925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec102c90 0x7f45ec0fa9e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0048d0 tx=0x7f45dc009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- 172.21.15.132:0/3165498925 shutdown_connections 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 --2- 172.21.15.132:0/3165498925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec102c90 0x7f45ec0fa9e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 --2- 172.21.15.132:0/3165498925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45ec1022f0 0x7f45ec1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- 172.21.15.132:0/3165498925 >> 172.21.15.132:0/3165498925 conn(0x7f45ec070a40 msgr2=0x7f45ec070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- 172.21.15.132:0/3165498925 shutdown_connections 2024-06-13T01:27:41.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- 172.21.15.132:0/3165498925 wait complete. 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 Processor -- start 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- start start 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 0x7f45ec1130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45ec102c90 0x7f45ec113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec113c30 con 0x7f45ec1022f0 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f3af3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec10d070 con 0x7f45ec102c90 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f188f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 0x7f45ec1130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f188f640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 0x7f45ec1130f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34512/0 (socket says 172.21.15.132:34512) 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.220+0000 7f45f188f640 1 -- 172.21.15.132:0/4006861656 learned_addr learned my addr 172.21.15.132:0/4006861656 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f108e640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45ec102c90 0x7f45ec113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f188f640 1 -- 172.21.15.132:0/4006861656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45ec102c90 msgr2=0x7f45ec113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f188f640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45ec102c90 0x7f45ec113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f188f640 1 -- 172.21.15.132:0/4006861656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc005040 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f188f640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 0x7f45ec1130f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f45e8002840 tx=0x7f45e8004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45e8007c30 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45e800f040 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec10d350 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45e800da30 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.221+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec10d840 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.222+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ec065510 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.223+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f45e80096a0 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.223+0000 7f45e27fc640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f45d8065430 0x7f45d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.223+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f45e8082330 con 0x7f45ec1022f0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.223+0000 7f45f108e640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f45d8065430 0x7f45d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:41.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.223+0000 7f45f108e640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f45d8065430 0x7f45d80678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f45dc000c00 tx=0x7f45dc035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:41.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.228+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f45e80507e0 con 0x7f45ec1022f0 2024-06-13T01:27:41.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.426+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45ec1094a0 con 0x7f45ec1022f0 2024-06-13T01:27:41.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.426+0000 7f45e27fc640 1 -- 172.21.15.132:0/4006861656 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f45e8053e00 con 0x7f45ec1022f0 2024-06-13T01:27:41.426 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:41.426 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f45d8065430 msgr2=0x7f45d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f45d8065430 0x7f45d80678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f45dc000c00 tx=0x7f45dc035040 comp rx=0 tx=0).stop 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 msgr2=0x7f45ec1130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 0x7f45ec1130f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f45e8002840 tx=0x7f45e8004570 comp rx=0 tx=0).stop 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 shutdown_connections 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f45d8065430 0x7f45d80678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45ec102c90 0x7f45ec113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 --2- 172.21.15.132:0/4006861656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45ec1022f0 0x7f45ec1130f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 >> 172.21.15.132:0/4006861656 conn(0x7f45ec070a40 msgr2=0x7f45ec1050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 shutdown_connections 2024-06-13T01:27:41.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:41.428+0000 7f45f3af3640 1 -- 172.21.15.132:0/4006861656 wait complete. 2024-06-13T01:27:41.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:41 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/4006861656' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:41.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:41 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/4006861656' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:42.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:42 smithi155 ceph-mon[35815]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:42.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:42 smithi132 ceph-mon[29330]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:44.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:44 smithi155 ceph-mon[35815]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:44 smithi132 ceph-mon[29330]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:45.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:45.342 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:46.671 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:46 smithi132 ceph-mon[29330]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:46.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.859+0000 7fb190d77640 1 -- 172.21.15.132:0/276508447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c1011b0 msgr2=0x7fb18c101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:46.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.859+0000 7fb190d77640 1 --2- 172.21.15.132:0/276508447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c1011b0 0x7fb18c101580 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0048d0 tx=0x7fb17c009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- 172.21.15.132:0/276508447 shutdown_connections 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 --2- 172.21.15.132:0/276508447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb18c101b50 0x7fb18c0fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 --2- 172.21.15.132:0/276508447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c1011b0 0x7fb18c101580 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- 172.21.15.132:0/276508447 >> 172.21.15.132:0/276508447 conn(0x7fb18c0f6d60 msgr2=0x7fb18c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- 172.21.15.132:0/276508447 shutdown_connections 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- 172.21.15.132:0/276508447 wait complete. 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 Processor -- start 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- start start 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb18c1011b0 0x7fb18c10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 0x7fb18c10e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c10eca0 con 0x7fb18c101b50 2024-06-13T01:27:46.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb190d77640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c10ee10 con 0x7fb18c1011b0 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb189d9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 0x7fb18c10e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb189d9b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 0x7fb18c10e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:43488/0 (socket says 172.21.15.132:43488) 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.860+0000 7fb189d9b640 1 -- 172.21.15.132:0/1875665315 learned_addr learned my addr 172.21.15.132:0/1875665315 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb18a59c640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb18c1011b0 0x7fb18c10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb189d9b640 1 -- 172.21.15.132:0/1875665315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb18c1011b0 msgr2=0x7fb18c10e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb189d9b640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb18c1011b0 0x7fb18c10e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb189d9b640 1 -- 172.21.15.132:0/1875665315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c005040 con 0x7fb18c101b50 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb189d9b640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 0x7fb18c10e5c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb18000b790 tx=0x7fb1800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb18000c700 con 0x7fb18c101b50 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18c1170f0 con 0x7fb18c101b50 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb18000cd30 con 0x7fb18c101b50 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb180013370 con 0x7fb18c101b50 2024-06-13T01:27:46.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.861+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c117610 con 0x7fb18c101b50 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.862+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c0fc920 con 0x7fb18c101b50 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.862+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb180012070 con 0x7fb18c101b50 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.863+0000 7fb17affd640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb174065430 0x7fb1740678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.863+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb1800820d0 con 0x7fb18c101b50 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.863+0000 7fb18a59c640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb174065430 0x7fb1740678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.863+0000 7fb18a59c640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb174065430 0x7fb1740678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb17c000c00 tx=0x7fb17c035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:46.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:46.867+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb180050580 con 0x7fb18c101b50 2024-06-13T01:27:46.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:46 smithi155 ceph-mon[35815]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:47.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.067+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb18c04b480 con 0x7fb18c101b50 2024-06-13T01:27:47.068 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.068+0000 7fb17affd640 1 -- 172.21.15.132:0/1875665315 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb180053ba0 con 0x7fb18c101b50 2024-06-13T01:27:47.068 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:47.068 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb174065430 msgr2=0x7fb1740678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb174065430 0x7fb1740678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb17c000c00 tx=0x7fb17c035040 comp rx=0 tx=0).stop 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 msgr2=0x7fb18c10e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 0x7fb18c10e5c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb18000b790 tx=0x7fb1800091d0 comp rx=0 tx=0).stop 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 shutdown_connections 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fb174065430 0x7fb1740678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb18c101b50 0x7fb18c10e5c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:47.070 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 --2- 172.21.15.132:0/1875665315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb18c1011b0 0x7fb18c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:47.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 >> 172.21.15.132:0/1875665315 conn(0x7fb18c0f6d60 msgr2=0x7fb18c0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:47.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 shutdown_connections 2024-06-13T01:27:47.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:47.070+0000 7fb190d77640 1 -- 172.21.15.132:0/1875665315 wait complete. 2024-06-13T01:27:47.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:47 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1875665315' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:47.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:47 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1875665315' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:48.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:48 smithi132 ceph-mon[29330]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:48.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:48 smithi155 ceph-mon[35815]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:49.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:49.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:27:50.712 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:50 smithi155 ceph-mon[35815]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:27:50.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:27:50.879 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:27:50.879 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:27:50.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:50 smithi132 ceph-mon[29330]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:50.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:27:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:27:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:27:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:27:50.968 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 -- 172.21.15.132:0/2632370436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 msgr2=0x7fcc4c0faa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2632370436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c0faa50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcc480048d0 tx=0x7fcc48009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 -- 172.21.15.132:0/2632370436 shutdown_connections 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2632370436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c0faa50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2632370436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 0x7fcc4c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 -- 172.21.15.132:0/2632370436 >> 172.21.15.132:0/2632370436 conn(0x7fcc4c070ab0 msgr2=0x7fcc4c070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 -- 172.21.15.132:0/2632370436 shutdown_connections 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 -- 172.21.15.132:0/2632370436 wait complete. 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 Processor -- start 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.481+0000 7fcc5367e640 1 -- start start 2024-06-13T01:27:52.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5367e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 0x7fcc4c113110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5367e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5367e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c113ce0 con 0x7fcc4c102d20 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5367e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c10d090 con 0x7fcc4c102380 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c113650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33482/0 (socket says 172.21.15.132:33482) 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 -- 172.21.15.132:0/2354458686 learned_addr learned my addr 172.21.15.132:0/2354458686 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5141a640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 0x7fcc4c113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 -- 172.21.15.132:0/2354458686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 msgr2=0x7fcc4c113110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 0x7fcc4c113110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 -- 172.21.15.132:0/2354458686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48005040 con 0x7fcc4c102d20 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5141a640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 0x7fcc4c113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:27:52.490 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc50c19640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c113650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcc48003080 tx=0x7fcc48009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc48038070 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc48008ee0 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c10d310 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.482+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc48033760 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.483+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c10d770 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.484+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c065520 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.484+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fcc48030020 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.484+0000 7fcc41ffb640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fcc38065430 0x7fcc380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.484+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcc480a1ff0 con 0x7fcc4c102d20 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.484+0000 7fcc5141a640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fcc38065430 0x7fcc380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.485+0000 7fcc5141a640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fcc38065430 0x7fcc380678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0099f0 tx=0x7fcc3c008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:52.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.488+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcc48070520 con 0x7fcc4c102d20 2024-06-13T01:27:52.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.688+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc4c0749d0 con 0x7fcc4c102d20 2024-06-13T01:27:52.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.688+0000 7fcc41ffb640 1 -- 172.21.15.132:0/2354458686 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcc48079020 con 0x7fcc4c102d20 2024-06-13T01:27:52.688 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:52.688 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fcc38065430 msgr2=0x7fcc380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fcc38065430 0x7fcc380678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0099f0 tx=0x7fcc3c008040 comp rx=0 tx=0).stop 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 msgr2=0x7fcc4c113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c113650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcc48003080 tx=0x7fcc48009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 shutdown_connections 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fcc38065430 0x7fcc380678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc4c102d20 0x7fcc4c113650 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 --2- 172.21.15.132:0/2354458686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc4c102380 0x7fcc4c113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:52.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 >> 172.21.15.132:0/2354458686 conn(0x7fcc4c070ab0 msgr2=0x7fcc4c0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:52.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 shutdown_connections 2024-06-13T01:27:52.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:52.690+0000 7fcc5367e640 1 -- 172.21.15.132:0/2354458686 wait complete. 2024-06-13T01:27:52.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:52 smithi155 ceph-mon[35815]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:52.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:52 smithi132 ceph-mon[29330]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:53.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:53 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2354458686' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:53.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:53 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2354458686' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:54.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:54 smithi155 ceph-mon[35815]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:54.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:54 smithi132 ceph-mon[29330]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:56.291 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:27:56.555 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:27:56.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:56 smithi155 ceph-mon[35815]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:56.903 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:56 smithi132 ceph-mon[29330]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:58.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 -- 172.21.15.132:0/1317715058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 msgr2=0x7efcd01016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/1317715058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd01016e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efcc00048d0 tx=0x7efcc0009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 -- 172.21.15.132:0/1317715058 shutdown_connections 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/1317715058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcd0101cb0 0x7efcd00fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/1317715058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd01016e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 -- 172.21.15.132:0/1317715058 >> 172.21.15.132:0/1317715058 conn(0x7efcd00f6d90 msgr2=0x7efcd00f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 -- 172.21.15.132:0/1317715058 shutdown_connections 2024-06-13T01:27:58.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.120+0000 7efcd6f6b640 1 -- 172.21.15.132:0/1317715058 wait complete. 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.121+0000 7efcd6f6b640 1 Processor -- start 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.121+0000 7efcd6f6b640 1 -- start start 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd6f6b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd010beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd6f6b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcd0101cb0 0x7efcd010c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd6f6b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd010c9f0 con 0x7efcd0101310 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd6f6b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd010cb60 con 0x7efcd0101cb0 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd010beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd010beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33492/0 (socket says 172.21.15.132:33492) 2024-06-13T01:27:58.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 -- 172.21.15.132:0/2903765545 learned_addr learned my addr 172.21.15.132:0/2903765545 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efccffff640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcd0101cb0 0x7efcd010c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 -- 172.21.15.132:0/2903765545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcd0101cb0 msgr2=0x7efcd010c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcd0101cb0 0x7efcd010c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 -- 172.21.15.132:0/2903765545 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0005040 con 0x7efcd0101310 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd4d07640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd010beb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efcc0003080 tx=0x7efcc0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcc0038070 con 0x7efcd0101310 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcc0008ee0 con 0x7efcd0101310 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcc003d870 con 0x7efcd0101310 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.122+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd0110f00 con 0x7efcd0101310 2024-06-13T01:27:58.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.123+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd0111440 con 0x7efcd0101310 2024-06-13T01:27:58.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.124+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd00fa9f0 con 0x7efcd0101310 2024-06-13T01:27:58.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.124+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7efcc0030020 con 0x7efcd0101310 2024-06-13T01:27:58.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.124+0000 7efccd7fa640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efcb8065430 0x7efcb80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:27:58.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.124+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efcc00a2240 con 0x7efcd0101310 2024-06-13T01:27:58.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.124+0000 7efccffff640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efcb8065430 0x7efcb80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:27:58.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.125+0000 7efccffff640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efcb8065430 0x7efcb80678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efcd00661d0 tx=0x7efcc4004330 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:27:58.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.128+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efcc00706d0 con 0x7efcd0101310 2024-06-13T01:27:58.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.335+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efcd0065fe0 con 0x7efcd0101310 2024-06-13T01:27:58.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.335+0000 7efccd7fa640 1 -- 172.21.15.132:0/2903765545 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7efcc0073cf0 con 0x7efcd0101310 2024-06-13T01:27:58.335 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:27:58.335 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efcb8065430 msgr2=0x7efcb80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efcb8065430 0x7efcb80678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efcd00661d0 tx=0x7efcc4004330 comp rx=0 tx=0).stop 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 msgr2=0x7efcd010beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd010beb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efcc0003080 tx=0x7efcc0009da0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 shutdown_connections 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efcb8065430 0x7efcb80678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcd0101cb0 0x7efcd010c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 --2- 172.21.15.132:0/2903765545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0101310 0x7efcd010beb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:27:58.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 >> 172.21.15.132:0/2903765545 conn(0x7efcd00f6d90 msgr2=0x7efcd00f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:27:58.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 shutdown_connections 2024-06-13T01:27:58.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:27:58.337+0000 7efcd6f6b640 1 -- 172.21.15.132:0/2903765545 wait complete. 2024-06-13T01:27:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:58 smithi155 ceph-mon[35815]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:58.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:27:58 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2903765545' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:27:58.956 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:58 smithi132 ceph-mon[29330]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:27:58.957 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:27:58 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2903765545' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:00.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:00 smithi155 ceph-mon[35815]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:00 smithi132 ceph-mon[29330]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:01.981 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:02.245 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:02.685 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:02 smithi132 ceph-mon[29330]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:02.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:02 smithi155 ceph-mon[35815]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:03.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 -- 172.21.15.132:0/633321210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc281011d0 msgr2=0x7efc281015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 --2- 172.21.15.132:0/633321210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc281011d0 0x7efc281015a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efc200048d0 tx=0x7efc20009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 -- 172.21.15.132:0/633321210 shutdown_connections 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 --2- 172.21.15.132:0/633321210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc28101b70 0x7efc280fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 --2- 172.21.15.132:0/633321210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc281011d0 0x7efc281015a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 -- 172.21.15.132:0/633321210 >> 172.21.15.132:0/633321210 conn(0x7efc280f6dc0 msgr2=0x7efc280f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 -- 172.21.15.132:0/633321210 shutdown_connections 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.787+0000 7efc3042b640 1 -- 172.21.15.132:0/633321210 wait complete. 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc3042b640 1 Processor -- start 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc3042b640 1 -- start start 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc3042b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc281011d0 0x7efc28113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:03.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc3042b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 0x7efc281135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc3042b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc28113c50 con 0x7efc28101b70 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc3042b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2810d000 con 0x7efc281011d0 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc2d9c6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 0x7efc281135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc2d9c6640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 0x7efc281135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44118/0 (socket says 172.21.15.132:44118) 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc2d9c6640 1 -- 172.21.15.132:0/765629020 learned_addr learned my addr 172.21.15.132:0/765629020 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc2d9c6640 1 -- 172.21.15.132:0/765629020 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc281011d0 msgr2=0x7efc28113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc2d9c6640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc281011d0 0x7efc28113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.788+0000 7efc2d9c6640 1 -- 172.21.15.132:0/765629020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc20005040 con 0x7efc28101b70 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.789+0000 7efc2d9c6640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 0x7efc281135c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00b790 tx=0x7efc1c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.789+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1c00c700 con 0x7efc28101b70 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.789+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2810d2e0 con 0x7efc28101b70 2024-06-13T01:28:03.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.789+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc1c00cd30 con 0x7efc28101b70 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.789+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1c013370 con 0x7efc28101b70 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.789+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2810d800 con 0x7efc28101b70 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.790+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc280fc940 con 0x7efc28101b70 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.790+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7efc1c012070 con 0x7efc28101b70 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.791+0000 7efc26ffd640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efc10065430 0x7efc100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.791+0000 7efc2e1c7640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efc10065430 0x7efc100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.791+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc1c0820d0 con 0x7efc28101b70 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.791+0000 7efc2e1c7640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efc10065430 0x7efc100678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc20032430 tx=0x7efc20009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:03.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.795+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efc1c050580 con 0x7efc28101b70 2024-06-13T01:28:03.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.992+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc280fa9c0 con 0x7efc28101b70 2024-06-13T01:28:03.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.993+0000 7efc26ffd640 1 -- 172.21.15.132:0/765629020 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7efc280fa9c0 con 0x7efc28101b70 2024-06-13T01:28:03.993 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:03.993 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efc10065430 msgr2=0x7efc100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efc10065430 0x7efc100678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc20032430 tx=0x7efc20009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 msgr2=0x7efc281135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 0x7efc281135c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00b790 tx=0x7efc1c0091d0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 shutdown_connections 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7efc10065430 0x7efc100678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc28101b70 0x7efc281135c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 --2- 172.21.15.132:0/765629020 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc281011d0 0x7efc28113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:03.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 >> 172.21.15.132:0/765629020 conn(0x7efc280f6dc0 msgr2=0x7efc280f8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:03.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 shutdown_connections 2024-06-13T01:28:03.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:03.995+0000 7efc3042b640 1 -- 172.21.15.132:0/765629020 wait complete. 2024-06-13T01:28:04.771 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:04 smithi132 ceph-mon[29330]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:04.772 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:04 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/765629020' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:04.772 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:04.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:04 smithi155 ceph-mon[35815]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:04.879 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:04 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/765629020' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:04.879 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:06.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:06 smithi155 ceph-mon[35815]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:06.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:06 smithi132 ceph-mon[29330]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:07.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:08.056 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:08.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:08 smithi155 ceph-mon[35815]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:08.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:08 smithi132 ceph-mon[29330]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:09.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.575+0000 7f7b221dc640 1 -- 172.21.15.132:0/1314099913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fbe80 msgr2=0x7f7b1c0fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:09.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.575+0000 7f7b221dc640 1 --2- 172.21.15.132:0/1314099913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fbe80 0x7f7b1c0fcd80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7b040048d0 tx=0x7f7b04009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.575+0000 7f7b221dc640 1 -- 172.21.15.132:0/1314099913 shutdown_connections 2024-06-13T01:28:09.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.575+0000 7f7b221dc640 1 --2- 172.21.15.132:0/1314099913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fbe80 0x7f7b1c0fcd80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.575+0000 7f7b221dc640 1 --2- 172.21.15.132:0/1314099913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.575+0000 7f7b221dc640 1 -- 172.21.15.132:0/1314099913 >> 172.21.15.132:0/1314099913 conn(0x7f7b1c072110 msgr2=0x7f7b1c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:09.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 -- 172.21.15.132:0/1314099913 shutdown_connections 2024-06-13T01:28:09.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 -- 172.21.15.132:0/1314099913 wait complete. 2024-06-13T01:28:09.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 Processor -- start 2024-06-13T01:28:09.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 -- start start 2024-06-13T01:28:09.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c10df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:09.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b1c0fbe80 0x7f7b1c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:09.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c10eb20 con 0x7f7b1c0fb570 2024-06-13T01:28:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b221dc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c10ec90 con 0x7f7b1c0fbe80 2024-06-13T01:28:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b1b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c10df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b1b7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c10df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:44140/0 (socket says 172.21.15.132:44140) 2024-06-13T01:28:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.576+0000 7f7b1b7fe640 1 -- 172.21.15.132:0/3490086568 learned_addr learned my addr 172.21.15.132:0/3490086568 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b13fff640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b1c0fbe80 0x7f7b1c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:09.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b1b7fe640 1 -- 172.21.15.132:0/3490086568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b1c0fbe80 msgr2=0x7f7b1c10e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:09.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b1b7fe640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b1c0fbe80 0x7f7b1c10e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b1b7fe640 1 -- 172.21.15.132:0/3490086568 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b04005040 con 0x7f7b1c0fb570 2024-06-13T01:28:09.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b1b7fe640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c10df90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c002a50 tx=0x7f7b0c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:09.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0c007c30 con 0x7f7b1c0fb570 2024-06-13T01:28:09.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b0c00f040 con 0x7f7b1c0fb570 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c117000 con 0x7f7b1c0fb570 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0c00db50 con 0x7f7b1c0fb570 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.577+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1c117510 con 0x7f7b1c0fb570 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.578+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1c0fc920 con 0x7f7b1c0fb570 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.578+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7b0c01c400 con 0x7f7b1c0fb570 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.579+0000 7f7b18ff9640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7b08065430 0x7f7b080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:09.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.579+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7b0c081d70 con 0x7f7b1c0fb570 2024-06-13T01:28:09.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.579+0000 7f7b13fff640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7b08065430 0x7f7b080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:09.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.579+0000 7f7b13fff640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7b08065430 0x7f7b080678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009f80 tx=0x7f7b04035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:09.604 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.583+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7b0c04cdf0 con 0x7f7b1c0fb570 2024-06-13T01:28:09.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.781+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b1c072c20 con 0x7f7b1c0fb570 2024-06-13T01:28:09.781 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.781+0000 7f7b18ff9640 1 -- 172.21.15.132:0/3490086568 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7b0c0502e0 con 0x7f7b1c0fb570 2024-06-13T01:28:09.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:09.782 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:09.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7b08065430 msgr2=0x7f7b080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:09.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7b08065430 0x7f7b080678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7b04009f80 tx=0x7f7b04035040 comp rx=0 tx=0).stop 2024-06-13T01:28:09.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 msgr2=0x7f7b1c10df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:09.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c10df90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c002a50 tx=0x7f7b0c004600 comp rx=0 tx=0).stop 2024-06-13T01:28:09.783 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 shutdown_connections 2024-06-13T01:28:09.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7b08065430 0x7f7b080678e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b1c0fbe80 0x7f7b1c10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 --2- 172.21.15.132:0/3490086568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b1c0fb570 0x7f7b1c10df90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:09.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 >> 172.21.15.132:0/3490086568 conn(0x7f7b1c072110 msgr2=0x7f7b1c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:09.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 shutdown_connections 2024-06-13T01:28:09.784 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:09.783+0000 7f7b221dc640 1 -- 172.21.15.132:0/3490086568 wait complete. 2024-06-13T01:28:10.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:10 smithi155 ceph-mon[35815]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:10.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:10 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3490086568' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:10.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:10 smithi132 ceph-mon[29330]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:10.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:10 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3490086568' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:12.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:12 smithi155 ceph-mon[35815]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:12.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:12 smithi132 ceph-mon[29330]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:13.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:13.717 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:14.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:14 smithi155 ceph-mon[35815]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:14.883 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:14 smithi132 ceph-mon[29330]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 -- 172.21.15.132:0/3858821689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fb570 msgr2=0x7f7c480fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 --2- 172.21.15.132:0/3858821689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fb570 0x7f7c480fb940 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7c380048d0 tx=0x7f7c38009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 -- 172.21.15.132:0/3858821689 shutdown_connections 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 --2- 172.21.15.132:0/3858821689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c480fbe80 0x7f7c480fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 --2- 172.21.15.132:0/3858821689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fb570 0x7f7c480fb940 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 -- 172.21.15.132:0/3858821689 >> 172.21.15.132:0/3858821689 conn(0x7f7c48072110 msgr2=0x7f7c48072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 -- 172.21.15.132:0/3858821689 shutdown_connections 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 -- 172.21.15.132:0/3858821689 wait complete. 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 Processor -- start 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 -- start start 2024-06-13T01:28:15.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.306+0000 7f7c5030d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c480fb570 0x7f7c48111f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c5030d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 0x7f7c481124b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c5030d640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48112b40 con 0x7f7c480fbe80 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c5030d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4810bf40 con 0x7f7c480fb570 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 0x7f7c481124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 0x7f7c481124b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38156/0 (socket says 172.21.15.132:38156) 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 -- 172.21.15.132:0/1165575623 learned_addr learned my addr 172.21.15.132:0/1165575623 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4e0a9640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c480fb570 0x7f7c48111f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 -- 172.21.15.132:0/1165575623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c480fb570 msgr2=0x7f7c48111f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c480fb570 0x7f7c48111f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 -- 172.21.15.132:0/1165575623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38005040 con 0x7f7c480fbe80 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c4d8a8640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 0x7f7c481124b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400daf0 tx=0x7f7c4400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c4400ed30 con 0x7f7c480fbe80 2024-06-13T01:28:15.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c4400ee90 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4810c1c0 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c44016a90 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.307+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4810c6e0 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.308+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c480fc920 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.309+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7c44014070 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.309+0000 7f7c3effd640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c34065430 0x7f7c340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.309+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7c44084670 con 0x7f7c480fbe80 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.309+0000 7f7c4e0a9640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c34065430 0x7f7c340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.310+0000 7f7c4e0a9640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c34065430 0x7f7c340678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009f80 tx=0x7f7c38035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:15.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.313+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7c44052ba0 con 0x7f7c480fbe80 2024-06-13T01:28:15.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.514+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c48072c20 con 0x7f7c480fbe80 2024-06-13T01:28:15.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.515+0000 7f7c3effd640 1 -- 172.21.15.132:0/1165575623 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7c440561c0 con 0x7f7c480fbe80 2024-06-13T01:28:15.516 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:15.516 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c34065430 msgr2=0x7f7c340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c34065430 0x7f7c340678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009f80 tx=0x7f7c38035040 comp rx=0 tx=0).stop 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 msgr2=0x7f7c481124b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 0x7f7c481124b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400daf0 tx=0x7f7c4400b460 comp rx=0 tx=0).stop 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 shutdown_connections 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7c34065430 0x7f7c340678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c480fbe80 0x7f7c481124b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 --2- 172.21.15.132:0/1165575623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c480fb570 0x7f7c48111f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 >> 172.21.15.132:0/1165575623 conn(0x7f7c48072110 msgr2=0x7f7c480f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:15.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 shutdown_connections 2024-06-13T01:28:15.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:15.517+0000 7f7c5030d640 1 -- 172.21.15.132:0/1165575623 wait complete. 2024-06-13T01:28:15.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:15 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1165575623' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:15.883 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:15 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1165575623' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:16.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:16 smithi155 ceph-mon[35815]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:16.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:16 smithi132 ceph-mon[29330]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:18.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:18 smithi132 ceph-mon[29330]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:18.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:18 smithi155 ceph-mon[35815]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:19.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:19.423 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:19.887 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:20.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:20.751 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:20 smithi132 ceph-mon[29330]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:20.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:20 smithi155 ceph-mon[35815]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 -- 172.21.15.132:0/4227144610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c0101b50 msgr2=0x7f50c00fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 --2- 172.21.15.132:0/4227144610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c0101b50 0x7f50c00fb1e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f50b00048d0 tx=0x7f50b0009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 -- 172.21.15.132:0/4227144610 shutdown_connections 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 --2- 172.21.15.132:0/4227144610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c0101b50 0x7f50c00fb1e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 --2- 172.21.15.132:0/4227144610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c01011b0 0x7f50c0101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 -- 172.21.15.132:0/4227144610 >> 172.21.15.132:0/4227144610 conn(0x7f50c00f6d60 msgr2=0x7f50c00f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 -- 172.21.15.132:0/4227144610 shutdown_connections 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.936+0000 7f50c857b640 1 -- 172.21.15.132:0/4227144610 wait complete. 2024-06-13T01:28:20.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c857b640 1 Processor -- start 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c857b640 1 -- start start 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c857b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 0x7f50c01152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c857b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c0101b50 0x7f50c0115800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c857b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c0115e90 con 0x7f50c01011b0 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c857b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c010f240 con 0x7f50c0101b50 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c6317640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 0x7f50c01152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c6317640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 0x7f50c01152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38172/0 (socket says 172.21.15.132:38172) 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c6317640 1 -- 172.21.15.132:0/1963579420 learned_addr learned my addr 172.21.15.132:0/1963579420 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c5b16640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c0101b50 0x7f50c0115800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c6317640 1 -- 172.21.15.132:0/1963579420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c0101b50 msgr2=0x7f50c0115800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c6317640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c0101b50 0x7f50c0115800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c6317640 1 -- 172.21.15.132:0/1963579420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0005040 con 0x7f50c01011b0 2024-06-13T01:28:20.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.937+0000 7f50c5b16640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c0101b50 0x7f50c0115800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.938+0000 7f50c6317640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 0x7f50c01152c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00b790 tx=0x7f50bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.938+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50bc00c700 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.938+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c010f520 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.938+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50bc00cd30 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.938+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50bc013370 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.938+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c010fa40 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.939+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c00fc920 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.939+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f50bc012070 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.940+0000 7f50b6ffd640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f50ac065430 0x7f50ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.940+0000 7f50c5b16640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f50ac065430 0x7f50ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.940+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f50bc0820d0 con 0x7f50c01011b0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.940+0000 7f50c5b16640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f50ac065430 0x7f50ac0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009f80 tx=0x7f50b0009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:20.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:20.944+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f50bc050580 con 0x7f50c01011b0 2024-06-13T01:28:21.144 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.144+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f50c004b4e0 con 0x7f50c01011b0 2024-06-13T01:28:21.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.144+0000 7f50b6ffd640 1 -- 172.21.15.132:0/1963579420 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f50bc053ba0 con 0x7f50c01011b0 2024-06-13T01:28:21.145 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:21.145 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:21.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f50ac065430 msgr2=0x7f50ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:21.146 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f50ac065430 0x7f50ac0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009f80 tx=0x7f50b0009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 msgr2=0x7f50c01152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 0x7f50c01152c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00b790 tx=0x7f50bc0091d0 comp rx=0 tx=0).stop 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 shutdown_connections 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f50ac065430 0x7f50ac0678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50c0101b50 0x7f50c0115800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 --2- 172.21.15.132:0/1963579420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50c01011b0 0x7f50c01152c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 >> 172.21.15.132:0/1963579420 conn(0x7f50c00f6d60 msgr2=0x7f50c00f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 shutdown_connections 2024-06-13T01:28:21.147 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:21.146+0000 7f50c857b640 1 -- 172.21.15.132:0/1963579420 wait complete. 2024-06-13T01:28:21.839 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:21 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1963579420' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:22.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:21 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1963579420' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:22.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:22 smithi132 ceph-mon[29330]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:23.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:22 smithi155 ceph-mon[35815]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:24.841 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:24.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:24 smithi132 ceph-mon[29330]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:25.105 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:25.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:24 smithi155 ceph-mon[35815]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:26.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 -- 172.21.15.132:0/1857710445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c101b50 msgr2=0x7f787c0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:26.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 --2- 172.21.15.132:0/1857710445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c101b50 0x7f787c0fb1e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f78640048d0 tx=0x7f7864009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 -- 172.21.15.132:0/1857710445 shutdown_connections 2024-06-13T01:28:26.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 --2- 172.21.15.132:0/1857710445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c101b50 0x7f787c0fb1e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 --2- 172.21.15.132:0/1857710445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f787c1011b0 0x7f787c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 -- 172.21.15.132:0/1857710445 >> 172.21.15.132:0/1857710445 conn(0x7f787c0f6d60 msgr2=0x7f787c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 -- 172.21.15.132:0/1857710445 shutdown_connections 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.600+0000 7f7880ce0640 1 -- 172.21.15.132:0/1857710445 wait complete. 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7880ce0640 1 Processor -- start 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7880ce0640 1 -- start start 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7880ce0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 0x7f787c10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7880ce0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f787c101b50 0x7f787c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7880ce0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c10ebb0 con 0x7f787c1011b0 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7880ce0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c10ed20 con 0x7f787c101b50 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f787a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 0x7f787c10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f787a59c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 0x7f787c10e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:60572/0 (socket says 172.21.15.132:60572) 2024-06-13T01:28:26.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f787a59c640 1 -- 172.21.15.132:0/59367219 learned_addr learned my addr 172.21.15.132:0/59367219 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f7879d9b640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f787c101b50 0x7f787c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.601+0000 7f787a59c640 1 -- 172.21.15.132:0/59367219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f787c101b50 msgr2=0x7f787c10e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f787a59c640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f787c101b50 0x7f787c10e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f787a59c640 1 -- 172.21.15.132:0/59367219 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7864005040 con 0x7f787c1011b0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f787a59c640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 0x7f787c10e020 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f786c00db00 tx=0x7f786c004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f786c017070 con 0x7f787c1011b0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f786c00f040 con 0x7f787c1011b0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c117090 con 0x7f787c1011b0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f786c003ae0 con 0x7f787c1011b0 2024-06-13T01:28:26.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.602+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c1175a0 con 0x7f787c1011b0 2024-06-13T01:28:26.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.603+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c0fc920 con 0x7f787c1011b0 2024-06-13T01:28:26.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.603+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f786c003c40 con 0x7f787c1011b0 2024-06-13T01:28:26.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.604+0000 7f7872ffd640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7868065430 0x7f78680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:26.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.604+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f786c0825c0 con 0x7f787c1011b0 2024-06-13T01:28:26.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.604+0000 7f7879d9b640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7868065430 0x7f78680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:26.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.604+0000 7f7879d9b640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7868065430 0x7f78680678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f78640047f0 tx=0x7f7864009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:26.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.608+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f786c050a70 con 0x7f787c1011b0 2024-06-13T01:28:26.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.807+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f787c0fa9a0 con 0x7f787c1011b0 2024-06-13T01:28:26.808 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:26 smithi132 ceph-mon[29330]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:26.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.808+0000 7f7872ffd640 1 -- 172.21.15.132:0/59367219 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f786c054090 con 0x7f787c1011b0 2024-06-13T01:28:26.808 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:26.809 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7868065430 msgr2=0x7f78680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7868065430 0x7f78680678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f78640047f0 tx=0x7f7864009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 msgr2=0x7f787c10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 0x7f787c10e020 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f786c00db00 tx=0x7f786c004e60 comp rx=0 tx=0).stop 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 shutdown_connections 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f7868065430 0x7f78680678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f787c101b50 0x7f787c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 --2- 172.21.15.132:0/59367219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c1011b0 0x7f787c10e020 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:26.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 >> 172.21.15.132:0/59367219 conn(0x7f787c0f6d60 msgr2=0x7f787c0f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:26.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 shutdown_connections 2024-06-13T01:28:26.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:26.810+0000 7f7880ce0640 1 -- 172.21.15.132:0/59367219 wait complete. 2024-06-13T01:28:27.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:26 smithi155 ceph-mon[35815]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:27.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:27 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/59367219' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:28.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:27 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/59367219' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:28.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:28 smithi132 ceph-mon[29330]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:28 smithi155 ceph-mon[35815]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:30.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:30.662 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:30.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:30 smithi132 ceph-mon[29330]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:31.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:30 smithi155 ceph-mon[35815]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:32.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 -- 172.21.15.132:0/238515935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0fd170 msgr2=0x7f656c0fd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 --2- 172.21.15.132:0/238515935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0fd170 0x7f656c0fd540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f65680048d0 tx=0x7f6568009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 -- 172.21.15.132:0/238515935 shutdown_connections 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 --2- 172.21.15.132:0/238515935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f656c0faa50 0x7f656c0faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 --2- 172.21.15.132:0/238515935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0fd170 0x7f656c0fd540 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 -- 172.21.15.132:0/238515935 >> 172.21.15.132:0/238515935 conn(0x7f656c070ab0 msgr2=0x7f656c070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 -- 172.21.15.132:0/238515935 shutdown_connections 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 -- 172.21.15.132:0/238515935 wait complete. 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.209+0000 7f6573d47640 1 Processor -- start 2024-06-13T01:28:32.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 -- start start 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 0x7f656c115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f656c0fd170 0x7f656c115790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c115d90 con 0x7f656c0faa50 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c10f1d0 con 0x7f656c0fd170 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 0x7f656c115250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 0x7f656c115250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:54678/0 (socket says 172.21.15.132:54678) 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 -- 172.21.15.132:0/2343882664 learned_addr learned my addr 172.21.15.132:0/2343882664 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 -- 172.21.15.132:0/2343882664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f656c0fd170 msgr2=0x7f656c115790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f656c0fd170 0x7f656c115790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 -- 172.21.15.132:0/2343882664 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568005040 con 0x7f656c0faa50 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6571ae3640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 0x7f656c115250 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6568003080 tx=0x7f6568009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:32.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6568038070 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6568008ee0 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f656803d870 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c10f450 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.210+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c10f910 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.211+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c0fba60 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.212+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6568030020 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.212+0000 7f65627fc640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6558065430 0x7f65580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.212+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f65680a2150 con 0x7f656c0faa50 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.212+0000 7f65712e2640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6558065430 0x7f65580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:32.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.213+0000 7f65712e2640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6558065430 0x7f65580678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f656c110b10 tx=0x7f655c007c20 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:32.219 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.217+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6568070720 con 0x7f656c0faa50 2024-06-13T01:28:32.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.419+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f656c1108c0 con 0x7f656c0faa50 2024-06-13T01:28:32.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.419+0000 7f65627fc640 1 -- 172.21.15.132:0/2343882664 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6568073d40 con 0x7f656c0faa50 2024-06-13T01:28:32.419 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:32.419 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6558065430 msgr2=0x7f65580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6558065430 0x7f65580678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f656c110b10 tx=0x7f655c007c20 comp rx=0 tx=0).stop 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 msgr2=0x7f656c115250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 0x7f656c115250 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6568003080 tx=0x7f6568009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 shutdown_connections 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6558065430 0x7f65580678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f656c0fd170 0x7f656c115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 --2- 172.21.15.132:0/2343882664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f656c0faa50 0x7f656c115250 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 >> 172.21.15.132:0/2343882664 conn(0x7f656c070ab0 msgr2=0x7f656c0f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.421+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 shutdown_connections 2024-06-13T01:28:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:32.422+0000 7f6573d47640 1 -- 172.21.15.132:0/2343882664 wait complete. 2024-06-13T01:28:32.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:32 smithi132 ceph-mon[29330]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:32.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:32 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2343882664' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:32.930 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:32 smithi155 ceph-mon[35815]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:32.931 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:32 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2343882664' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:34.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:34 smithi132 ceph-mon[29330]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:34.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:34 smithi155 ceph-mon[35815]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:35.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:36.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:36.332 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:36.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:36 smithi132 ceph-mon[29330]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:37.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:36 smithi155 ceph-mon[35815]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 -- 172.21.15.132:0/1293938424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 msgr2=0x7f302c0701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 --2- 172.21.15.132:0/1293938424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c0701c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f301c0048d0 tx=0x7f301c009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 -- 172.21.15.132:0/1293938424 shutdown_connections 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 --2- 172.21.15.132:0/1293938424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c0701c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 --2- 172.21.15.132:0/1293938424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f302c071440 0x7f302c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 -- 172.21.15.132:0/1293938424 >> 172.21.15.132:0/1293938424 conn(0x7f302c0f7e80 msgr2=0x7f302c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 -- 172.21.15.132:0/1293938424 shutdown_connections 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.840+0000 7f30339d7640 1 -- 172.21.15.132:0/1293938424 wait complete. 2024-06-13T01:28:37.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f30339d7640 1 Processor -- start 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f30339d7640 1 -- start start 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f30339d7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c119660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f30339d7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f302c071440 0x7f302c119bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f30339d7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c11a210 con 0x7f302c06fd50 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f30339d7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c113600 con 0x7f302c071440 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c119660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c119660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:54704/0 (socket says 172.21.15.132:54704) 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 -- 172.21.15.132:0/2552453988 learned_addr learned my addr 172.21.15.132:0/2552453988 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3030f72640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f302c071440 0x7f302c119bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 -- 172.21.15.132:0/2552453988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f302c071440 msgr2=0x7f302c119bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:37.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f302c071440 0x7f302c119bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 -- 172.21.15.132:0/2552453988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c005040 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.841+0000 7f3031773640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c119660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3028002a50 tx=0x7f3028004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.842+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3028007c30 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.842+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f302800f040 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.842+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c1138e0 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.842+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302800dbe0 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.842+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c113e00 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.843+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c070f50 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.843+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f302800dd40 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.843+0000 7f3021ffb640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3018065430 0x7f30180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.844+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3028082470 con 0x7f302c06fd50 2024-06-13T01:28:37.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.844+0000 7f3030f72640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3018065430 0x7f30180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:37.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.844+0000 7f3030f72640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3018065430 0x7f30180678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f302c114f20 tx=0x7f301c035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:37.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:37.848+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3028050920 con 0x7f302c06fd50 2024-06-13T01:28:38.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.047+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f302c10f9d0 con 0x7f302c06fd50 2024-06-13T01:28:38.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.047+0000 7f3021ffb640 1 -- 172.21.15.132:0/2552453988 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3028053f40 con 0x7f302c06fd50 2024-06-13T01:28:38.047 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:38.047 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:38.049 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.049+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3018065430 msgr2=0x7f30180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.049+0000 7f30339d7640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3018065430 0x7f30180678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f302c114f20 tx=0x7f301c035040 comp rx=0 tx=0).stop 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.049+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 msgr2=0x7f302c119660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.049+0000 7f30339d7640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c119660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3028002a50 tx=0x7f3028004600 comp rx=0 tx=0).stop 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 shutdown_connections 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f3018065430 0x7f30180678e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f302c071440 0x7f302c119bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 --2- 172.21.15.132:0/2552453988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f302c06fd50 0x7f302c119660 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 >> 172.21.15.132:0/2552453988 conn(0x7f302c0f7e80 msgr2=0x7f302c0f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 shutdown_connections 2024-06-13T01:28:38.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:38.050+0000 7f30339d7640 1 -- 172.21.15.132:0/2552453988 wait complete. 2024-06-13T01:28:38.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:38 smithi132 ceph-mon[29330]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:38.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:38 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2552453988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:39.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:38 smithi155 ceph-mon[35815]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:39.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:38 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2552453988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:40.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:40 smithi132 ceph-mon[29330]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:41.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:40 smithi155 ceph-mon[35815]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:41.642 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:41.906 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:42.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:42 smithi132 ceph-mon[29330]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:43.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:42 smithi155 ceph-mon[35815]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:43.447 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.445+0000 7facd4670640 1 -- 172.21.15.132:0/1554177081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc065010 msgr2=0x7faccc0653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.445+0000 7facd4670640 1 --2- 172.21.15.132:0/1554177081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc065010 0x7faccc0653e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7facc80048d0 tx=0x7facc8009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- 172.21.15.132:0/1554177081 shutdown_connections 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 --2- 172.21.15.132:0/1554177081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc0fdea0 0x7faccc0fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 --2- 172.21.15.132:0/1554177081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc065010 0x7faccc0653e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- 172.21.15.132:0/1554177081 >> 172.21.15.132:0/1554177081 conn(0x7faccc072110 msgr2=0x7faccc072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- 172.21.15.132:0/1554177081 shutdown_connections 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- 172.21.15.132:0/1554177081 wait complete. 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 Processor -- start 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- start start 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc065010 0x7faccc112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 0x7faccc113510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:43.448 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc113b10 con 0x7faccc0fdea0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd4670640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc113c80 con 0x7faccc065010 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd1c0b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 0x7faccc113510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd1c0b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 0x7faccc113510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40210/0 (socket says 172.21.15.132:40210) 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.446+0000 7facd1c0b640 1 -- 172.21.15.132:0/3132462988 learned_addr learned my addr 172.21.15.132:0/3132462988 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd240c640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc065010 0x7faccc112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd1c0b640 1 -- 172.21.15.132:0/3132462988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc065010 msgr2=0x7faccc112fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd1c0b640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc065010 0x7faccc112fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd1c0b640 1 -- 172.21.15.132:0/3132462988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc8005040 con 0x7faccc0fdea0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd1c0b640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 0x7faccc113510 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7facbc002a50 tx=0x7facbc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facbc007c30 con 0x7faccc0fdea0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facbc00f040 con 0x7faccc0fdea0 2024-06-13T01:28:43.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc10d100 con 0x7faccc0fdea0 2024-06-13T01:28:43.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facbc00db60 con 0x7faccc0fdea0 2024-06-13T01:28:43.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.447+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc10d6a0 con 0x7faccc0fdea0 2024-06-13T01:28:43.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.448+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc0fda40 con 0x7faccc0fdea0 2024-06-13T01:28:43.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.448+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7facbc00dcc0 con 0x7faccc0fdea0 2024-06-13T01:28:43.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.449+0000 7facc2ffd640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7facb8065430 0x7facb80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:43.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.449+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7facbc082310 con 0x7faccc0fdea0 2024-06-13T01:28:43.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.449+0000 7facd240c640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7facb8065430 0x7facb80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:43.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.449+0000 7facd240c640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7facb8065430 0x7facb80678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7facc8009f80 tx=0x7facc8035040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:43.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.453+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7facbc0507c0 con 0x7faccc0fdea0 2024-06-13T01:28:43.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.650+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faccc072c20 con 0x7faccc0fdea0 2024-06-13T01:28:43.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.650+0000 7facc2ffd640 1 -- 172.21.15.132:0/3132462988 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7facbc053de0 con 0x7faccc0fdea0 2024-06-13T01:28:43.651 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:43.651 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.652+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7facb8065430 msgr2=0x7facb80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.652+0000 7facd4670640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7facb8065430 0x7facb80678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7facc8009f80 tx=0x7facc8035040 comp rx=0 tx=0).stop 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.652+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 msgr2=0x7faccc113510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.652+0000 7facd4670640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 0x7faccc113510 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7facbc002a50 tx=0x7facbc004600 comp rx=0 tx=0).stop 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 shutdown_connections 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7facb8065430 0x7facb80678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faccc0fdea0 0x7faccc113510 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 --2- 172.21.15.132:0/3132462988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc065010 0x7faccc112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 >> 172.21.15.132:0/3132462988 conn(0x7faccc072110 msgr2=0x7faccc0f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:43.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 shutdown_connections 2024-06-13T01:28:43.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:43.653+0000 7facd4670640 1 -- 172.21.15.132:0/3132462988 wait complete. 2024-06-13T01:28:43.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:43 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3132462988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:44.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:43 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3132462988' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:44.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:44 smithi132 ceph-mon[29330]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:45.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:44 smithi155 ceph-mon[35815]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:46.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:46 smithi132 ceph-mon[29330]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:47.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:46 smithi155 ceph-mon[35815]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:47.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:47.607 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:48.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:48 smithi132 ceph-mon[29330]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:49.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:48 smithi155 ceph-mon[35815]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:49.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 -- 172.21.15.132:0/3425853579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fbe80 msgr2=0x7fab100fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:49.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 --2- 172.21.15.132:0/3425853579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fbe80 0x7fab100fcd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faaf80048d0 tx=0x7faaf8009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 -- 172.21.15.132:0/3425853579 shutdown_connections 2024-06-13T01:28:49.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 --2- 172.21.15.132:0/3425853579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fbe80 0x7fab100fcd80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 --2- 172.21.15.132:0/3425853579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fb570 0x7fab100fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 -- 172.21.15.132:0/3425853579 >> 172.21.15.132:0/3425853579 conn(0x7fab10072110 msgr2=0x7fab10072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 -- 172.21.15.132:0/3425853579 shutdown_connections 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.226+0000 7fab16928640 1 -- 172.21.15.132:0/3425853579 wait complete. 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab16928640 1 Processor -- start 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab16928640 1 -- start start 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab16928640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 0x7fab10111f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:49.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab16928640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fbe80 0x7fab10112490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:49.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab16928640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab10112b20 con 0x7fab100fb570 2024-06-13T01:28:49.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab16928640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1010bed0 con 0x7fab100fbe80 2024-06-13T01:28:49.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab0ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 0x7fab10111f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:49.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab0ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 0x7fab10111f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:40226/0 (socket says 172.21.15.132:40226) 2024-06-13T01:28:49.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.227+0000 7fab0ffff640 1 -- 172.21.15.132:0/710562462 learned_addr learned my addr 172.21.15.132:0/710562462 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:49.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0f7fe640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fbe80 0x7fab10112490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:49.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0ffff640 1 -- 172.21.15.132:0/710562462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fbe80 msgr2=0x7fab10112490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:49.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0ffff640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fbe80 0x7fab10112490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0ffff640 1 -- 172.21.15.132:0/710562462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf8005040 con 0x7fab100fb570 2024-06-13T01:28:49.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0f7fe640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fbe80 0x7fab10112490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-13T01:28:49.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0ffff640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 0x7fab10111f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b790 tx=0x7fab000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab0000c700 con 0x7fab100fb570 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab0000cd30 con 0x7fab100fb570 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1010c1b0 con 0x7fab100fb570 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab00013370 con 0x7fab100fb570 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.228+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1010c6d0 con 0x7fab100fb570 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.229+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab100fc920 con 0x7fab100fb570 2024-06-13T01:28:49.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.230+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fab00012070 con 0x7fab100fb570 2024-06-13T01:28:49.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.230+0000 7fab0cff9640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faafc065430 0x7faafc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:49.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.230+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fab00081f00 con 0x7fab100fb570 2024-06-13T01:28:49.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.230+0000 7fab0f7fe640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faafc065430 0x7faafc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:49.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.230+0000 7fab0f7fe640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faafc065430 0x7faafc0678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009f80 tx=0x7faaf8009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:49.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.234+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fab000503b0 con 0x7fab100fb570 2024-06-13T01:28:49.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.435+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fab1010c340 con 0x7fab100fb570 2024-06-13T01:28:49.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.435+0000 7fab0cff9640 1 -- 172.21.15.132:0/710562462 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fab1010c340 con 0x7fab100fb570 2024-06-13T01:28:49.436 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:49.436 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:49.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faafc065430 msgr2=0x7faafc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:49.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faafc065430 0x7faafc0678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009f80 tx=0x7faaf8009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 msgr2=0x7fab10111f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 0x7fab10111f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b790 tx=0x7fab000091d0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 shutdown_connections 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7faafc065430 0x7faafc0678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab100fbe80 0x7fab10112490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 --2- 172.21.15.132:0/710562462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab100fb570 0x7fab10111f50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 >> 172.21.15.132:0/710562462 conn(0x7fab10072110 msgr2=0x7fab100f8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 shutdown_connections 2024-06-13T01:28:49.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:49.438+0000 7fab16928640 1 -- 172.21.15.132:0/710562462 wait complete. 2024-06-13T01:28:49.697 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:49 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:49.697 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:49 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/710562462' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:49 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:28:50.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:49 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/710562462' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:50 smithi132 ceph-mon[29330]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:28:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:28:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:28:50.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:50 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:28:51.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:50 smithi155 ceph-mon[35815]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:51.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-13T01:28:51.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-13T01:28:51.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-13T01:28:51.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:50 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:28:52.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:52 smithi132 ceph-mon[29330]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:53.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:52 smithi155 ceph-mon[35815]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:53.162 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:53.426 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:28:54.828 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:54 smithi132 ceph-mon[29330]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:55.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- 172.21.15.132:0/1139426343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 msgr2=0x7f2914101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:55.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 --2- 172.21.15.132:0/1139426343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f2914101580 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f29040048d0 tx=0x7f2904009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- 172.21.15.132:0/1139426343 shutdown_connections 2024-06-13T01:28:55.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 --2- 172.21.15.132:0/1139426343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2914101b50 0x7f29140fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 --2- 172.21.15.132:0/1139426343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f2914101580 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- 172.21.15.132:0/1139426343 >> 172.21.15.132:0/1139426343 conn(0x7f29140f6d60 msgr2=0x7f29140f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- 172.21.15.132:0/1139426343 shutdown_connections 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- 172.21.15.132:0/1139426343 wait complete. 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 Processor -- start 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- start start 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f291410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2914101b50 0x7f291410e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291410ec30 con 0x7f29141011b0 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.042+0000 7f2918b09640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291410eda0 con 0x7f2914101b50 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f291410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f291410e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59530/0 (socket says 172.21.15.132:59530) 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 -- 172.21.15.132:0/567833595 learned_addr learned my addr 172.21.15.132:0/567833595 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:28:55.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f2911d9b640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2914101b50 0x7f291410e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 -- 172.21.15.132:0/567833595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2914101b50 msgr2=0x7f291410e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2914101b50 0x7f291410e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 -- 172.21.15.132:0/567833595 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2904005040 con 0x7f29141011b0 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f291259c640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f291410e060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2904003080 tx=0x7f2904009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2904038070 con 0x7f29141011b0 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2904008ee0 con 0x7f29141011b0 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29141170d0 con 0x7f29141011b0 2024-06-13T01:28:55.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f290403d870 con 0x7f29141011b0 2024-06-13T01:28:55.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.043+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2914117540 con 0x7f29141011b0 2024-06-13T01:28:55.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.044+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29140fc920 con 0x7f29141011b0 2024-06-13T01:28:55.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.045+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2904030020 con 0x7f29141011b0 2024-06-13T01:28:55.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.045+0000 7f2902ffd640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f28fc065430 0x7f28fc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:28:55.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.045+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29040a2180 con 0x7f29141011b0 2024-06-13T01:28:55.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.045+0000 7f2911d9b640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f28fc065430 0x7f28fc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:28:55.048 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.046+0000 7f2911d9b640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f28fc065430 0x7f28fc0678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f29140661d0 tx=0x7f2908009780 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:28:55.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.049+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f29040706b0 con 0x7f29141011b0 2024-06-13T01:28:55.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:54 smithi155 ceph-mon[35815]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:55.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.247+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f29141083a0 con 0x7f29141011b0 2024-06-13T01:28:55.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.248+0000 7f2902ffd640 1 -- 172.21.15.132:0/567833595 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2904073cd0 con 0x7f29141011b0 2024-06-13T01:28:55.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:28:55.249 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:28:55.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f28fc065430 msgr2=0x7f28fc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:55.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f28fc065430 0x7f28fc0678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f29140661d0 tx=0x7f2908009780 comp rx=0 tx=0).stop 2024-06-13T01:28:55.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 msgr2=0x7f291410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:28:55.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f291410e060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2904003080 tx=0x7f2904009da0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 shutdown_connections 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f28fc065430 0x7f28fc0678e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2914101b50 0x7f291410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 --2- 172.21.15.132:0/567833595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29141011b0 0x7f291410e060 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 >> 172.21.15.132:0/567833595 conn(0x7f29140f6d60 msgr2=0x7f29140f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 shutdown_connections 2024-06-13T01:28:55.252 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:28:55.250+0000 7f2918b09640 1 -- 172.21.15.132:0/567833595 wait complete. 2024-06-13T01:28:55.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:55 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/567833595' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:55.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:55 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/567833595' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:28:56.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:56 smithi132 ceph-mon[29330]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:57.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:56 smithi155 ceph-mon[35815]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:58.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:28:58 smithi132 ceph-mon[29330]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:58.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:28:59.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:28:58 smithi155 ceph-mon[35815]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:28:59.250 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:00.805 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 -- 172.21.15.132:0/2913260906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 msgr2=0x7f64e01026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 --2- 172.21.15.132:0/2913260906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e01026c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f64d00048d0 tx=0x7f64d0009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 -- 172.21.15.132:0/2913260906 shutdown_connections 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 --2- 172.21.15.132:0/2913260906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 0x7f64e00fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 --2- 172.21.15.132:0/2913260906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e01026c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 -- 172.21.15.132:0/2913260906 >> 172.21.15.132:0/2913260906 conn(0x7f64e0070a40 msgr2=0x7f64e0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 -- 172.21.15.132:0/2913260906 shutdown_connections 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.799+0000 7f64e731c640 1 -- 172.21.15.132:0/2913260906 wait complete. 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e731c640 1 Processor -- start 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e731c640 1 -- start start 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e731c640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e0113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:00.806 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e731c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 0x7f64e01136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e731c640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0113d40 con 0x7f64e01022f0 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e731c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e010d0f0 con 0x7f64e0102c90 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e0113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e0113170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59544/0 (socket says 172.21.15.132:59544) 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 -- 172.21.15.132:0/4257799670 learned_addr learned my addr 172.21.15.132:0/4257799670 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e48b7640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 0x7f64e01136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 -- 172.21.15.132:0/4257799670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 msgr2=0x7f64e01136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 0x7f64e01136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 -- 172.21.15.132:0/4257799670 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0005040 con 0x7f64e01022f0 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e48b7640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 0x7f64e01136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.800+0000 7f64e50b8640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e0113170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f64d0008fc0 tx=0x7f64d0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.801+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64d0038070 con 0x7f64e01022f0 2024-06-13T01:29:00.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.801+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64d0032b80 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.801+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e010d370 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.801+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64d003dc20 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.801+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e010d830 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.802+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e0065510 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.802+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f64d0030020 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.802+0000 7f64d5ffb640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f64cc065430 0x7f64cc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.803+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f64d00a2e20 con 0x7f64e01022f0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.803+0000 7f64e48b7640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f64cc065430 0x7f64cc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:00.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.803+0000 7f64e48b7640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f64cc065430 0x7f64cc0678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f64e010ea30 tx=0x7f64dc008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:00.809 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:00.807+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f64d0071410 con 0x7f64e01022f0 2024-06-13T01:29:00.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:00 smithi132 ceph-mon[29330]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.005+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f64e004b510 con 0x7f64e01022f0 2024-06-13T01:29:01.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.005+0000 7f64d5ffb640 1 -- 172.21.15.132:0/4257799670 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f64d0074a30 con 0x7f64e01022f0 2024-06-13T01:29:01.006 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:01.006 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:01.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f64cc065430 msgr2=0x7f64cc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f64cc065430 0x7f64cc0678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f64e010ea30 tx=0x7f64dc008040 comp rx=0 tx=0).stop 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 msgr2=0x7f64e0113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e0113170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f64d0008fc0 tx=0x7f64d0009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 shutdown_connections 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f64cc065430 0x7f64cc0678e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e0102c90 0x7f64e01136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 --2- 172.21.15.132:0/4257799670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64e01022f0 0x7f64e0113170 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 >> 172.21.15.132:0/4257799670 conn(0x7f64e0070a40 msgr2=0x7f64e01050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 shutdown_connections 2024-06-13T01:29:01.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:01.007+0000 7f64e731c640 1 -- 172.21.15.132:0/4257799670 wait complete. 2024-06-13T01:29:01.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:00 smithi155 ceph-mon[35815]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:01.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:01 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/4257799670' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:02.030 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:01 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/4257799670' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:02.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:02 smithi132 ceph-mon[29330]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:03.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:02 smithi155 ceph-mon[35815]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:04.584 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:04.868 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:04.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:04 smithi132 ceph-mon[29330]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:04.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:04 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:29:05.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:04 smithi155 ceph-mon[35815]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:05.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:04 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:29:06.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 -- 172.21.15.132:0/3203600302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 msgr2=0x7f4514102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:06.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3203600302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514102670 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f45040048d0 tx=0x7f4504009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 -- 172.21.15.132:0/3203600302 shutdown_connections 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3203600302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4514102c40 0x7f45140fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3203600302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514102670 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 -- 172.21.15.132:0/3203600302 >> 172.21.15.132:0/3203600302 conn(0x7f45140f7e50 msgr2=0x7f45140fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 -- 172.21.15.132:0/3203600302 shutdown_connections 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 -- 172.21.15.132:0/3203600302 wait complete. 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 Processor -- start 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 -- start start 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.505+0000 7f4519ee7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4514102c40 0x7f45141135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4519ee7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514113c30 con 0x7f45141022a0 2024-06-13T01:29:06.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4519ee7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451410cfe0 con 0x7f4514102c40 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514113060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53902/0 (socket says 172.21.15.132:53902) 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 -- 172.21.15.132:0/3603096287 learned_addr learned my addr 172.21.15.132:0/3603096287 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f450bfff640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4514102c40 0x7f45141135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 -- 172.21.15.132:0/3603096287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4514102c40 msgr2=0x7f45141135a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4514102c40 0x7f45141135a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 -- 172.21.15.132:0/3603096287 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4504005040 con 0x7f45141022a0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f45137fe640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514113060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4504003080 tx=0x7f4504009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4504038070 con 0x7f45141022a0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451410d260 con 0x7f45141022a0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4504008ee0 con 0x7f45141022a0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f450403d870 con 0x7f45141022a0 2024-06-13T01:29:06.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.506+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451410d720 con 0x7f45141022a0 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.507+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45140fda10 con 0x7f45141022a0 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.508+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4504030020 con 0x7f45141022a0 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.508+0000 7f4510ff9640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4500065430 0x7f45000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.508+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f45040a2180 con 0x7f45141022a0 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.508+0000 7f450bfff640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4500065430 0x7f45000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.509+0000 7f450bfff640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4500065430 0x7f45000678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f451410e920 tx=0x7f44fc009120 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:06.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.512+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f45040706b0 con 0x7f45141022a0 2024-06-13T01:29:06.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.709+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f451404b4b0 con 0x7f45141022a0 2024-06-13T01:29:06.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.709+0000 7f4510ff9640 1 -- 172.21.15.132:0/3603096287 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4504073cd0 con 0x7f45141022a0 2024-06-13T01:29:06.710 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:06.710 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:06.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4500065430 msgr2=0x7f45000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:06.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4500065430 0x7f45000678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f451410e920 tx=0x7f44fc009120 comp rx=0 tx=0).stop 2024-06-13T01:29:06.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 msgr2=0x7f4514113060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514113060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4504003080 tx=0x7f4504009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 shutdown_connections 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f4500065430 0x7f45000678e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4514102c40 0x7f45141135a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 --2- 172.21.15.132:0/3603096287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45141022a0 0x7f4514113060 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 >> 172.21.15.132:0/3603096287 conn(0x7f45140f7e50 msgr2=0x7f45140f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 shutdown_connections 2024-06-13T01:29:06.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:06.711+0000 7f4519ee7640 1 -- 172.21.15.132:0/3603096287 wait complete. 2024-06-13T01:29:06.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:06 smithi132 ceph-mon[29330]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:06.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:06 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3603096287' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:07.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:06 smithi155 ceph-mon[35815]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:07.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:06 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3603096287' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:09.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:08 smithi155 ceph-mon[35815]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:09.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:08 smithi132 ceph-mon[29330]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:10.333 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:10.598 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:11.069 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:10 smithi132 ceph-mon[29330]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:11.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:10 smithi155 ceph-mon[35815]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 -- 172.21.15.132:0/3293478026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c1001d0 msgr2=0x7fc38c1005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3293478026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c1001d0 0x7fc38c1005a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc3880048d0 tx=0x7fc388009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 -- 172.21.15.132:0/3293478026 shutdown_connections 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3293478026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c0fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3293478026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c1001d0 0x7fc38c1005a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 -- 172.21.15.132:0/3293478026 >> 172.21.15.132:0/3293478026 conn(0x7fc38c070a40 msgr2=0x7fc38c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 -- 172.21.15.132:0/3293478026 shutdown_connections 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.279+0000 7fc3933c4640 1 -- 172.21.15.132:0/3293478026 wait complete. 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc3933c4640 1 Processor -- start 2024-06-13T01:29:12.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc3933c4640 1 -- start start 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc3933c4640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc3933c4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc38c1001d0 0x7fc38c10e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc3933c4640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c10ec10 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc3933c4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c10ed80 con 0x7fc38c1001d0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc391160640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc391160640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34298/0 (socket says 172.21.15.132:34298) 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc391160640 1 -- 172.21.15.132:0/3101739679 learned_addr learned my addr 172.21.15.132:0/3101739679 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc39095f640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc38c1001d0 0x7fc38c10e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc391160640 1 -- 172.21.15.132:0/3101739679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc38c1001d0 msgr2=0x7fc38c10e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc391160640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc38c1001d0 0x7fc38c10e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.280+0000 7fc391160640 1 -- 172.21.15.132:0/3101739679 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388005040 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.281+0000 7fc391160640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c10e080 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc3880043d0 tx=0x7fc388009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.281+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc388038070 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.281+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc388032970 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.281+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c1170f0 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.281+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc38803da60 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.281+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c1174d0 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.282+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c0fcb00 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.282+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc388030020 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.283+0000 7fc381ffb640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc378065430 0x7fc3780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.283+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc3880a2550 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.283+0000 7fc39095f640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc378065430 0x7fc3780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.283+0000 7fc39095f640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc378065430 0x7fc3780678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009af0 tx=0x7fc37c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:12.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.287+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc388070a00 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.485+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc38c0fb190 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.485+0000 7fc381ffb640 1 -- 172.21.15.132:0/3101739679 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc388044300 con 0x7fc38c0fbaf0 2024-06-13T01:29:12.485 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:12.486 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:12.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc378065430 msgr2=0x7fc3780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:12.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc378065430 0x7fc3780678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009af0 tx=0x7fc37c00b040 comp rx=0 tx=0).stop 2024-06-13T01:29:12.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 msgr2=0x7fc38c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c10e080 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc3880043d0 tx=0x7fc388009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 shutdown_connections 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7fc378065430 0x7fc3780678e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc38c1001d0 0x7fc38c10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 --2- 172.21.15.132:0/3101739679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc38c0fbaf0 0x7fc38c10e080 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 >> 172.21.15.132:0/3101739679 conn(0x7fc38c070a40 msgr2=0x7fc38c0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 shutdown_connections 2024-06-13T01:29:12.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:12.487+0000 7fc3933c4640 1 -- 172.21.15.132:0/3101739679 wait complete. 2024-06-13T01:29:12.938 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:12 smithi132 ceph-mon[29330]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:12.939 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:12 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3101739679' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:13.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:12 smithi155 ceph-mon[35815]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:13.129 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:12 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3101739679' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:15.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:14 smithi155 ceph-mon[35815]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:15.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:14 smithi132 ceph-mon[29330]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:16.333 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:16.593 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:17.042 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:16 smithi132 ceph-mon[29330]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:17.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:16 smithi155 ceph-mon[35815]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 -- 172.21.15.132:0/422176148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 msgr2=0x7f94040fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 --2- 172.21.15.132:0/422176148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f94040fb1e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f93f80048d0 tx=0x7f93f8009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 -- 172.21.15.132:0/422176148 shutdown_connections 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 --2- 172.21.15.132:0/422176148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f94040fb1e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 --2- 172.21.15.132:0/422176148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94041011b0 0x7f9404101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 -- 172.21.15.132:0/422176148 >> 172.21.15.132:0/422176148 conn(0x7f94040f6d60 msgr2=0x7f94040f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:18.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 -- 172.21.15.132:0/422176148 shutdown_connections 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.150+0000 7f9409d03640 1 -- 172.21.15.132:0/422176148 wait complete. 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9409d03640 1 Processor -- start 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9409d03640 1 -- start start 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9409d03640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94041011b0 0x7f9404111ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9409d03640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f9404112430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9409d03640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9404112a30 con 0x7f9404101b50 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9409d03640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940410be70 con 0x7f94041011b0 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9402ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f9404112430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9402ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f9404112430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34316/0 (socket says 172.21.15.132:34316) 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9402ffd640 1 -- 172.21.15.132:0/1190933060 learned_addr learned my addr 172.21.15.132:0/1190933060 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:18.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f94037fe640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94041011b0 0x7f9404111ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9402ffd640 1 -- 172.21.15.132:0/1190933060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94041011b0 msgr2=0x7f9404111ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9402ffd640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94041011b0 0x7f9404111ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.151+0000 7f9402ffd640 1 -- 172.21.15.132:0/1190933060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8005040 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.152+0000 7f9402ffd640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f9404112430 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f93f8003080 tx=0x7f93f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.152+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93f8038070 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.152+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93f8008ee0 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.152+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940410c0f0 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.152+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93f803d870 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.152+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940410c630 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.153+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94040fc920 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.153+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f93f8030020 con 0x7f9404101b50 2024-06-13T01:29:18.160 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.154+0000 7f93ebfff640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f93ec065430 0x7f93ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:18.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.154+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f93f80a21e0 con 0x7f9404101b50 2024-06-13T01:29:18.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.154+0000 7f94037fe640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f93ec065430 0x7f93ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:18.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.154+0000 7f94037fe640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f93ec065430 0x7f93ec0678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f93f4007fd0 tx=0x7f93f4009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:18.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.158+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f93f8070710 con 0x7f9404101b50 2024-06-13T01:29:18.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.356+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f94041082f0 con 0x7f9404101b50 2024-06-13T01:29:18.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.356+0000 7f93ebfff640 1 -- 172.21.15.132:0/1190933060 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f93f8073d30 con 0x7f9404101b50 2024-06-13T01:29:18.356 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:18.357 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f93ec065430 msgr2=0x7f93ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f93ec065430 0x7f93ec0678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f93f4007fd0 tx=0x7f93f4009040 comp rx=0 tx=0).stop 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 msgr2=0x7f9404112430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f9404112430 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f93f8003080 tx=0x7f93f8009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 shutdown_connections 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f93ec065430 0x7f93ec0678e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9404101b50 0x7f9404112430 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 --2- 172.21.15.132:0/1190933060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94041011b0 0x7f9404111ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 >> 172.21.15.132:0/1190933060 conn(0x7f94040f6d60 msgr2=0x7f94040f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 shutdown_connections 2024-06-13T01:29:18.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:18.358+0000 7f9409d03640 1 -- 172.21.15.132:0/1190933060 wait complete. 2024-06-13T01:29:18.996 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:18 smithi132 ceph-mon[29330]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:18.996 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:18 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/1190933060' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:19.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:18 smithi155 ceph-mon[35815]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:19.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:18 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/1190933060' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:20.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:19 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:29:20.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:19 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:29:21.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:20 smithi155 ceph-mon[35815]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:21.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:20 smithi132 ceph-mon[29330]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:22.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:22.287 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:23.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:22 smithi155 ceph-mon[35815]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:23.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:22 smithi132 ceph-mon[29330]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:23.909 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 -- 172.21.15.132:0/3355307124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 msgr2=0x7f29301016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 --2- 172.21.15.132:0/3355307124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301016e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f29200048d0 tx=0x7f2920009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 -- 172.21.15.132:0/3355307124 shutdown_connections 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 --2- 172.21.15.132:0/3355307124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2930101cb0 0x7f29300fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 --2- 172.21.15.132:0/3355307124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301016e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 -- 172.21.15.132:0/3355307124 >> 172.21.15.132:0/3355307124 conn(0x7f29300f6d90 msgr2=0x7f29300f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 -- 172.21.15.132:0/3355307124 shutdown_connections 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.907+0000 7f29358c7640 1 -- 172.21.15.132:0/3355307124 wait complete. 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f29358c7640 1 Processor -- start 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f29358c7640 1 -- start start 2024-06-13T01:29:23.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f29358c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301120c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f29358c7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2930101cb0 0x7f2930112600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f29358c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2930112c90 con 0x7f2930101310 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f29358c7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293010c040 con 0x7f2930101cb0 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f292effd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f292effd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301120c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51092/0 (socket says 172.21.15.132:51092) 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.908+0000 7f292effd640 1 -- 172.21.15.132:0/914927761 learned_addr learned my addr 172.21.15.132:0/914927761 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f2927fff640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2930101cb0 0x7f2930112600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f292effd640 1 -- 172.21.15.132:0/914927761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2930101cb0 msgr2=0x7f2930112600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f292effd640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2930101cb0 0x7f2930112600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f292effd640 1 -- 172.21.15.132:0/914927761 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2920005040 con 0x7f2930101310 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f292effd640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301120c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2920003080 tx=0x7f2920009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:23.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2920038070 con 0x7f2930101310 2024-06-13T01:29:23.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293010c2c0 con 0x7f2930101310 2024-06-13T01:29:23.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2920008ee0 con 0x7f2930101310 2024-06-13T01:29:23.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f292003d870 con 0x7f2930101310 2024-06-13T01:29:23.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.909+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293010c780 con 0x7f2930101310 2024-06-13T01:29:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.910+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29300fa9f0 con 0x7f2930101310 2024-06-13T01:29:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.911+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2920030020 con 0x7f2930101310 2024-06-13T01:29:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.911+0000 7f29277fe640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f291c065430 0x7f291c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.911+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29200a20c0 con 0x7f2930101310 2024-06-13T01:29:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.911+0000 7f2927fff640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f291c065430 0x7f291c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:23.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.911+0000 7f2927fff640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f291c065430 0x7f291c0678e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f293010d980 tx=0x7f2918009120 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:23.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:23.915+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2920070690 con 0x7f2930101310 2024-06-13T01:29:24.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.117+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f29301084d0 con 0x7f2930101310 2024-06-13T01:29:24.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.117+0000 7f29277fe640 1 -- 172.21.15.132:0/914927761 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2920073cb0 con 0x7f2930101310 2024-06-13T01:29:24.117 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:24.117 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:24.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f291c065430 msgr2=0x7f291c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f291c065430 0x7f291c0678e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f293010d980 tx=0x7f2918009120 comp rx=0 tx=0).stop 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 msgr2=0x7f29301120c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301120c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2920003080 tx=0x7f2920009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 shutdown_connections 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f291c065430 0x7f291c0678e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2930101cb0 0x7f2930112600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 --2- 172.21.15.132:0/914927761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2930101310 0x7f29301120c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.119+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 >> 172.21.15.132:0/914927761 conn(0x7f29300f6d90 msgr2=0x7f29300f7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.120+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 shutdown_connections 2024-06-13T01:29:24.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:24.120+0000 7f29358c7640 1 -- 172.21.15.132:0/914927761 wait complete. 2024-06-13T01:29:25.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:24 smithi155 ceph-mon[35815]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:25.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:24 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/914927761' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:25.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:24 smithi132 ceph-mon[29330]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:25.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:24 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/914927761' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:27.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:26 smithi155 ceph-mon[35815]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:27.211 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:26 smithi132 ceph-mon[29330]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:27.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:28.121 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:28.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:28 smithi132 ceph-mon[29330]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:29.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:28 smithi155 ceph-mon[35815]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:29.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 -- 172.21.15.132:0/2113550132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 msgr2=0x7f6388102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2113550132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388102690 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f63780048d0 tx=0x7f6378009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 -- 172.21.15.132:0/2113550132 shutdown_connections 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2113550132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6388102c60 0x7f63880fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2113550132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388102690 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 -- 172.21.15.132:0/2113550132 >> 172.21.15.132:0/2113550132 conn(0x7f63880f7e50 msgr2=0x7f63880fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 -- 172.21.15.132:0/2113550132 shutdown_connections 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.664+0000 7f638d5a1640 1 -- 172.21.15.132:0/2113550132 wait complete. 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f638d5a1640 1 Processor -- start 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f638d5a1640 1 -- start start 2024-06-13T01:29:29.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f638d5a1640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f638d5a1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6388102c60 0x7f6388113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f638d5a1640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388113e70 con 0x7f63881022c0 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f638d5a1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638810d2b0 con 0x7f6388102c60 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.665+0000 7f6386ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f6386ffd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:51120/0 (socket says 172.21.15.132:51120) 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f6386ffd640 1 -- 172.21.15.132:0/2477243654 learned_addr learned my addr 172.21.15.132:0/2477243654 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f637e7fc640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6388102c60 0x7f6388113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f6386ffd640 1 -- 172.21.15.132:0/2477243654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6388102c60 msgr2=0x7f6388113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f6386ffd640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6388102c60 0x7f6388113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f6386ffd640 1 -- 172.21.15.132:0/2477243654 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6378005040 con 0x7f63881022c0 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f6386ffd640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388113330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f63780043d0 tx=0x7f6378009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:29.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6378038070 con 0x7f63881022c0 2024-06-13T01:29:29.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6378032970 con 0x7f63881022c0 2024-06-13T01:29:29.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638810d530 con 0x7f63881022c0 2024-06-13T01:29:29.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f637803da60 con 0x7f63881022c0 2024-06-13T01:29:29.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.666+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638810da70 con 0x7f63881022c0 2024-06-13T01:29:29.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.667+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63880fda30 con 0x7f63881022c0 2024-06-13T01:29:29.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.668+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6378030020 con 0x7f63881022c0 2024-06-13T01:29:29.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.668+0000 7f637ffff640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6368065430 0x7f63680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:29.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.668+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f63780a25c0 con 0x7f63881022c0 2024-06-13T01:29:29.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.668+0000 7f637e7fc640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6368065430 0x7f63680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:29.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.669+0000 7f637e7fc640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6368065430 0x7f63680678e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f638810ebf0 tx=0x7f637000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:29.672 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.672+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6378070a70 con 0x7f63881022c0 2024-06-13T01:29:29.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.870+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f638804b480 con 0x7f63881022c0 2024-06-13T01:29:29.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.870+0000 7f637ffff640 1 -- 172.21.15.132:0/2477243654 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6378074090 con 0x7f63881022c0 2024-06-13T01:29:29.871 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:29.871 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:29.872 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6368065430 msgr2=0x7f63680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6368065430 0x7f63680678e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f638810ebf0 tx=0x7f637000b040 comp rx=0 tx=0).stop 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 msgr2=0x7f6388113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388113330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f63780043d0 tx=0x7f6378009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 shutdown_connections 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6368065430 0x7f63680678e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6388102c60 0x7f6388113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 --2- 172.21.15.132:0/2477243654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63881022c0 0x7f6388113330 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.872+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 >> 172.21.15.132:0/2477243654 conn(0x7f63880f7e50 msgr2=0x7f63880f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.873+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 shutdown_connections 2024-06-13T01:29:29.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:29.873+0000 7f638d5a1640 1 -- 172.21.15.132:0/2477243654 wait complete. 2024-06-13T01:29:30.961 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:30 smithi132 ceph-mon[29330]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:30.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:30 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/2477243654' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:31.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:30 smithi155 ceph-mon[35815]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:31.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:30 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/2477243654' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:33.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:32 smithi155 ceph-mon[35815]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:33.212 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:32 smithi132 ceph-mon[29330]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:33.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:33.904 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:34.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:34 smithi132 ceph-mon[29330]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:34.962 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:34 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:29:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:34 smithi155 ceph-mon[35815]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:35.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:34 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 -- 172.21.15.132:0/4157350981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c101b50 msgr2=0x7ff22c0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 --2- 172.21.15.132:0/4157350981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c101b50 0x7ff22c0fb1e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0048d0 tx=0x7ff21c009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 -- 172.21.15.132:0/4157350981 shutdown_connections 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 --2- 172.21.15.132:0/4157350981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c101b50 0x7ff22c0fb1e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 --2- 172.21.15.132:0/4157350981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff22c1011b0 0x7ff22c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 -- 172.21.15.132:0/4157350981 >> 172.21.15.132:0/4157350981 conn(0x7ff22c0f6d60 msgr2=0x7ff22c0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 -- 172.21.15.132:0/4157350981 shutdown_connections 2024-06-13T01:29:35.571 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.546+0000 7ff2337ee640 1 -- 172.21.15.132:0/4157350981 wait complete. 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff2337ee640 1 Processor -- start 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff2337ee640 1 -- start start 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff2337ee640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 0x7ff22c112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff2337ee640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff22c101b50 0x7ff22c113510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff2337ee640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c113b10 con 0x7ff22c1011b0 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff2337ee640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c113c80 con 0x7ff22c101b50 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 0x7ff22c112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 0x7ff22c112fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:35930/0 (socket says 172.21.15.132:35930) 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 -- 172.21.15.132:0/449954721 learned_addr learned my addr 172.21.15.132:0/449954721 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 -- 172.21.15.132:0/449954721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff22c101b50 msgr2=0x7ff22c113510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff22c101b50 0x7ff22c113510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.572 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 -- 172.21.15.132:0/449954721 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c005040 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.547+0000 7ff23158a640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 0x7ff22c112fd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c810 tx=0x7ff2280043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.548+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22800a820 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.548+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff22800ae50 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.548+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22c10d100 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.548+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2280115c0 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.548+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22c10d6a0 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.549+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22c0fc920 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.549+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff22800a980 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.549+0000 7ff221ffb640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff218065430 0x7ff2180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.549+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff228082a60 con 0x7ff22c1011b0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.549+0000 7ff230d89640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff218065430 0x7ff2180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:35.573 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.550+0000 7ff230d89640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff218065430 0x7ff2180678e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff21c032430 tx=0x7ff21c009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:35.574 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.553+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff228050f10 con 0x7ff22c1011b0 2024-06-13T01:29:35.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.754+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff22c0fa9a0 con 0x7ff22c1011b0 2024-06-13T01:29:35.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.754+0000 7ff221ffb640 1 -- 172.21.15.132:0/449954721 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff228054530 con 0x7ff22c1011b0 2024-06-13T01:29:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:35.755 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-13T01:29:35.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff218065430 msgr2=0x7ff2180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff218065430 0x7ff2180678e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff21c032430 tx=0x7ff21c009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 msgr2=0x7ff22c112fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 0x7ff22c112fd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c810 tx=0x7ff2280043c0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 shutdown_connections 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7ff218065430 0x7ff2180678e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff22c101b50 0x7ff22c113510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 --2- 172.21.15.132:0/449954721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff22c1011b0 0x7ff22c112fd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 >> 172.21.15.132:0/449954721 conn(0x7ff22c0f6d60 msgr2=0x7ff22c0657b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 shutdown_connections 2024-06-13T01:29:35.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:35.756+0000 7ff2337ee640 1 -- 172.21.15.132:0/449954721 wait complete. 2024-06-13T01:29:36.628 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:36 smithi155 ceph-mon[35815]: from='client.14624 -' entity='client.crash.smithi132' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:29:36.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:36 smithi155 ceph-mon[35815]: from='client.14628 -' entity='client.crash.smithi132' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:29:36.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:36 smithi155 ceph-mon[35815]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:29:36.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:36 smithi155 ceph-mon[35815]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:36.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:36 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/449954721' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:36.691 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:36 smithi132 ceph-mon[29330]: from='client.14624 -' entity='client.crash.smithi132' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:29:36.691 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:36 smithi132 ceph-mon[29330]: from='client.14628 -' entity='client.crash.smithi132' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-13T01:29:36.691 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:36 smithi132 ceph-mon[29330]: from='mgr.14210 172.21.15.132:0/4050233138' entity='mgr.smithi132.vjweoa' 2024-06-13T01:29:36.691 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:36 smithi132 ceph-mon[29330]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:36.692 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:36 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/449954721' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:37.628 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:37 smithi155 ceph-mon[35815]: Health check update: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-13T01:29:37.629 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:37 smithi132 ceph-mon[29330]: Health check update: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-13T01:29:38.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:38 smithi132 ceph-mon[29330]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:38.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:38 smithi155 ceph-mon[35815]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:39.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 -- ceph health --format=json 2024-06-13T01:29:39.707 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/mon.smithi132/config 2024-06-13T01:29:40.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:40 smithi132 ceph-mon[29330]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:40.878 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:40 smithi155 ceph-mon[35815]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:41.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 -- 172.21.15.132:0/1079057528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd40fa930 msgr2=0x7f6cd40fad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:41.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 --2- 172.21.15.132:0/1079057528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd40fa930 0x7f6cd40fad80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40048d0 tx=0x7f6cc4009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 -- 172.21.15.132:0/1079057528 shutdown_connections 2024-06-13T01:29:41.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 --2- 172.21.15.132:0/1079057528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd40fa930 0x7f6cd40fad80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 --2- 172.21.15.132:0/1079057528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cd4102360 0x7f6cd4102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.315 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 -- 172.21.15.132:0/1079057528 >> 172.21.15.132:0/1079057528 conn(0x7f6cd4071820 msgr2=0x7f6cd4071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 -- 172.21.15.132:0/1079057528 shutdown_connections 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.313+0000 7f6cdc092640 1 -- 172.21.15.132:0/1079057528 wait complete. 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cdc092640 1 Processor -- start 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cdc092640 1 -- start start 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cdc092640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cd40fa930 0x7f6cd41130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cdc092640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 0x7f6cd4113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cdc092640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4113c90 con 0x7f6cd4102360 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cdc092640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd410d040 con 0x7f6cd40fa930 2024-06-13T01:29:41.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 0x7f6cd4113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 0x7f6cd4113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:35946/0 (socket says 172.21.15.132:35946) 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 -- 172.21.15.132:0/3161082567 learned_addr learned my addr 172.21.15.132:0/3161082567 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd9e2e640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cd40fa930 0x7f6cd41130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 -- 172.21.15.132:0/3161082567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cd40fa930 msgr2=0x7f6cd41130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cd40fa930 0x7f6cd41130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 -- 172.21.15.132:0/3161082567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4005040 con 0x7f6cd4102360 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.314+0000 7f6cd962d640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 0x7f6cd4113600 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4032430 tx=0x7f6cc4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.315+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc4038070 con 0x7f6cd4102360 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.315+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd410d2c0 con 0x7f6cd4102360 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.315+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc4008ee0 con 0x7f6cd4102360 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.315+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc403d7a0 con 0x7f6cd4102360 2024-06-13T01:29:41.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.315+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd410d780 con 0x7f6cd4102360 2024-06-13T01:29:41.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.316+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd40fdc50 con 0x7f6cd4102360 2024-06-13T01:29:41.318 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.316+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6cc4030020 con 0x7f6cd4102360 2024-06-13T01:29:41.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.316+0000 7f6cca7fc640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6cc0065430 0x7f6cc00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-13T01:29:41.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.317+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6cc40a2ae0 con 0x7f6cd4102360 2024-06-13T01:29:41.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.317+0000 7f6cd9e2e640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6cc0065430 0x7f6cc00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-13T01:29:41.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.317+0000 7f6cd9e2e640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6cc0065430 0x7f6cc00678e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0007f30 tx=0x7f6cd0007420 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-13T01:29:41.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.321+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6cc4071050 con 0x7f6cd4102360 2024-06-13T01:29:41.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.521+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6cd4072330 con 0x7f6cd4102360 2024-06-13T01:29:41.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.521+0000 7f6cca7fc640 1 -- 172.21.15.132:0/3161082567 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6cc4074670 con 0x7f6cd4102360 2024-06-13T01:29:41.521 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-06-13T01:29:41.522 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6cc0065430 msgr2=0x7f6cc00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6cc0065430 0x7f6cc00678e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0007f30 tx=0x7f6cd0007420 comp rx=0 tx=0).stop 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 msgr2=0x7f6cd4113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 0x7f6cd4113600 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4032430 tx=0x7f6cc4009da0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 shutdown_connections 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:6800/2992514013,v1:172.21.15.132:6801/2992514013] conn(0x7f6cc0065430 0x7f6cc00678e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cd4102360 0x7f6cd4113600 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 --2- 172.21.15.132:0/3161082567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cd40fa930 0x7f6cd41130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 >> 172.21.15.132:0/3161082567 conn(0x7f6cd4071820 msgr2=0x7f6cd40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 shutdown_connections 2024-06-13T01:29:41.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-06-13T01:29:41.523+0000 7f6cdc092640 1 -- 172.21.15.132:0/3161082567 wait complete. 2024-06-13T01:29:42.157 INFO:tasks.cephadm:Teardown begin 2024-06-13T01:29:42.157 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-13T01:29:42.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:29:42.188 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:29:42.222 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-13T01:29:42.222 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-13T01:29:42.243 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-13T01:29:42.280 INFO:tasks.cephadm:Stopping all daemons... 2024-06-13T01:29:42.280 INFO:tasks.cephadm.mon.smithi132:Stopping mon.smithi132... 2024-06-13T01:29:42.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132 2024-06-13T01:29:42.385 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:42 smithi132 systemd[1]: Stopping Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:29:42.628 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:42 smithi155 ceph-mon[35815]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:42.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:42 smithi155 ceph-mon[35815]: from='client.? 172.21.15.132:0/3161082567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:42.711 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:42 smithi132 ceph-mon[29330]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-13T01:29:42.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:42 smithi132 ceph-mon[29330]: from='client.? 172.21.15.132:0/3161082567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-13T01:29:42.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:42 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132[29326]: 2024-06-13T01:29:42.568+0000 7f8f28a51640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi132 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:29:42.712 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:42 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132[29326]: 2024-06-13T01:29:42.568+0000 7f8f28a51640 -1 mon.smithi132@0(leader) e2 *** Got Signal Terminated *** 2024-06-13T01:29:43.087 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:42 smithi132 podman[80447]: 2024-06-13 01:29:42.823412702 +0000 UTC m=+0.349174431 container died e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, org.label-schema.build-date=20240612) 2024-06-13T01:29:43.461 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:43 smithi132 podman[80447]: 2024-06-13 01:29:43.244386192 +0000 UTC m=+0.770147960 container cleanup e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-296b94f, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-06-13T01:29:43.462 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:43 smithi132 bash[80447]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132 2024-06-13T01:29:43.819 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service' 2024-06-13T01:29:43.840 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:43 smithi132 podman[80540]: 2024-06-13 01:29:43.581299732 +0000 UTC m=+0.756774308 container remove e4c0e9cea28c81089e982e4823b824bfde1f6a003e782d8f9b7c60a5ee8ac4ef (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi132, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240612) 2024-06-13T01:29:43.841 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:43 smithi132 systemd[1]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service: Deactivated successfully. 2024-06-13T01:29:43.841 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:43 smithi132 systemd[1]: Stopped Ceph mon.smithi132 for ad8d4854-2922-11ef-bc9c-c7b262605968. 2024-06-13T01:29:43.841 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Jun 13 01:29:43 smithi132 systemd[1]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi132.service: Consumed 8.093s CPU time. 2024-06-13T01:29:43.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:29:43.866 INFO:tasks.cephadm.mon.smithi132:Stopped mon.smithi132 2024-06-13T01:29:43.866 INFO:tasks.cephadm.mon.smithi155:Stopping mon.smithi155... 2024-06-13T01:29:43.867 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi155 2024-06-13T01:29:44.163 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:43 smithi155 systemd[1]: Stopping Ceph mon.smithi155 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:29:44.628 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:44 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi155[35767]: 2024-06-13T01:29:44.163+0000 7fc0b458d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi155 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:29:44.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:44 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi155[35767]: 2024-06-13T01:29:44.163+0000 7fc0b458d640 -1 mon.smithi155@1(peon) e2 *** Got Signal Terminated *** 2024-06-13T01:29:44.629 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:44 smithi155 podman[59489]: 2024-06-13 01:29:44.399383241 +0000 UTC m=+0.331495798 container died ce2e9cee7e8adef952dd5d4229e3d6f5ddc5df4e5deee2c40149cfb5143496d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi155, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-13T01:29:45.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:44 smithi155 podman[59489]: 2024-06-13 01:29:44.863544236 +0000 UTC m=+0.795656792 container cleanup ce2e9cee7e8adef952dd5d4229e3d6f5ddc5df4e5deee2c40149cfb5143496d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi155, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-13T01:29:45.128 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jun 13 01:29:44 smithi155 bash[59489]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-mon-smithi155 2024-06-13T01:29:45.288 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@mon.smithi155.service' 2024-06-13T01:29:45.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:29:45.338 INFO:tasks.cephadm.mon.smithi155:Stopped mon.smithi155 2024-06-13T01:29:45.338 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-13T01:29:45.338 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.0 2024-06-13T01:29:45.657 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:45 smithi132 systemd[1]: Stopping Ceph osd.0 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:29:45.961 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:45 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0[47104]: 2024-06-13T01:29:45.657+0000 7fcb0924c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:29:45.962 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:45 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0[47104]: 2024-06-13T01:29:45.657+0000 7fcb0924c640 -1 osd.0 40 *** Got signal Terminated *** 2024-06-13T01:29:45.962 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:45 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0[47104]: 2024-06-13T01:29:45.657+0000 7fcb0924c640 -1 osd.0 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T01:29:51.462 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:51 smithi132 podman[80665]: 2024-06-13 01:29:51.193555402 +0000 UTC m=+5.658876543 container died 3cd2ca995f364c48eaef1c9d7ecee90dca4ef8ebb81be42288c9a3d9ed9a8258 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-296b94f, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-06-13T01:29:51.908 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:51 smithi132 podman[80665]: 2024-06-13 01:29:51.649548582 +0000 UTC m=+6.114869699 container cleanup 3cd2ca995f364c48eaef1c9d7ecee90dca4ef8ebb81be42288c9a3d9ed9a8258 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-296b94f, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-13T01:29:51.909 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:51 smithi132 bash[80665]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0 2024-06-13T01:29:52.212 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:51 smithi132 podman[80681]: 2024-06-13 01:29:51.909306003 +0000 UTC m=+0.709965474 container remove 3cd2ca995f364c48eaef1c9d7ecee90dca4ef8ebb81be42288c9a3d9ed9a8258 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-13T01:29:52.503 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:52 smithi132 podman[80769]: 2024-06-13 01:29:52.21850151 +0000 UTC m=+0.091250442 container create 61171aa9c3ef81ef0bdc322d418765daf9cc9b418e53337c64a15c854ecfc87c (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-296b94f, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-13T01:29:52.504 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:52 smithi132 podman[80769]: 2024-06-13 01:29:52.144665872 +0000 UTC m=+0.017414804 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56 2024-06-13T01:29:52.504 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:52 smithi132 podman[80769]: 2024-06-13 01:29:52.421069349 +0000 UTC m=+0.293818286 container init 61171aa9c3ef81ef0bdc322d418765daf9cc9b418e53337c64a15c854ecfc87c (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T01:29:52.504 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:52 smithi132 podman[80769]: 2024-06-13 01:29:52.424385171 +0000 UTC m=+0.297134104 container start 61171aa9c3ef81ef0bdc322d418765daf9cc9b418e53337c64a15c854ecfc87c (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T01:29:52.504 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:52 smithi132 podman[80769]: 2024-06-13 01:29:52.504436726 +0000 UTC m=+0.377185682 container attach 61171aa9c3ef81ef0bdc322d418765daf9cc9b418e53337c64a15c854ecfc87c (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-296b94f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T01:29:52.962 INFO:journalctl@ceph.osd.0.smithi132.stdout:Jun 13 01:29:52 smithi132 podman[80769]: 2024-06-13 01:29:52.666191376 +0000 UTC m=+0.538940308 container died 61171aa9c3ef81ef0bdc322d418765daf9cc9b418e53337c64a15c854ecfc87c (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=quincy-296b94f) 2024-06-13T01:29:53.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.0.service' 2024-06-13T01:29:53.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:29:53.349 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-13T01:29:53.349 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-13T01:29:53.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.1 2024-06-13T01:29:53.695 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:53 smithi132 systemd[1]: Stopping Ceph osd.1 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:29:53.962 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:53 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1[54461]: 2024-06-13T01:29:53.694+0000 7f24099b4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:29:53.962 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:53 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1[54461]: 2024-06-13T01:29:53.694+0000 7f24099b4640 -1 osd.1 40 *** Got signal Terminated *** 2024-06-13T01:29:53.962 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:53 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1[54461]: 2024-06-13T01:29:53.694+0000 7f24099b4640 -1 osd.1 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T01:29:59.428 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:59 smithi132 podman[80872]: 2024-06-13 01:29:59.168360001 +0000 UTC m=+5.591410140 container died 043cdc0c2fa6f18dcdeb366354c4af918df8695259ca67b1e859037f29f9a5ce (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-296b94f, org.label-schema.schema-version=1.0) 2024-06-13T01:29:59.713 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:59 smithi132 podman[80872]: 2024-06-13 01:29:59.611421176 +0000 UTC m=+6.034471377 container cleanup 043cdc0c2fa6f18dcdeb366354c4af918df8695259ca67b1e859037f29f9a5ce (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-296b94f, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T01:29:59.714 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:59 smithi132 bash[80872]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1 2024-06-13T01:30:00.211 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:29:59 smithi132 podman[80887]: 2024-06-13 01:29:59.887855179 +0000 UTC m=+0.721939146 container remove 043cdc0c2fa6f18dcdeb366354c4af918df8695259ca67b1e859037f29f9a5ce (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-296b94f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-06-13T01:30:00.507 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:30:00 smithi132 podman[80977]: 2024-06-13 01:30:00.143134567 +0000 UTC m=+0.017438371 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56 2024-06-13T01:30:00.507 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:30:00 smithi132 podman[80977]: 2024-06-13 01:30:00.24692253 +0000 UTC m=+0.121226317 container create 519d454d73e4be8f4b4859be8ed99e91b9eb6e09a16d0be1674b3a0bb666819b (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True) 2024-06-13T01:30:00.507 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:30:00 smithi132 podman[80977]: 2024-06-13 01:30:00.50834177 +0000 UTC m=+0.382645557 container init 519d454d73e4be8f4b4859be8ed99e91b9eb6e09a16d0be1674b3a0bb666819b (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1-deactivate, RELEASE=quincy-296b94f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T01:30:00.962 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:30:00 smithi132 podman[80977]: 2024-06-13 01:30:00.512143983 +0000 UTC m=+0.386447772 container start 519d454d73e4be8f4b4859be8ed99e91b9eb6e09a16d0be1674b3a0bb666819b (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-13T01:30:00.962 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:30:00 smithi132 podman[80977]: 2024-06-13 01:30:00.591646669 +0000 UTC m=+0.465950469 container attach 519d454d73e4be8f4b4859be8ed99e91b9eb6e09a16d0be1674b3a0bb666819b (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-13T01:30:00.962 INFO:journalctl@ceph.osd.1.smithi132.stdout:Jun 13 01:30:00 smithi132 podman[80977]: 2024-06-13 01:30:00.753636142 +0000 UTC m=+0.627939930 container died 519d454d73e4be8f4b4859be8ed99e91b9eb6e09a16d0be1674b3a0bb666819b (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-1-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T01:30:01.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.1.service' 2024-06-13T01:30:01.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:30:01.523 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-13T01:30:01.523 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-13T01:30:01.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.2 2024-06-13T01:30:01.761 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:01 smithi132 systemd[1]: Stopping Ceph osd.2 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:30:02.211 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:01 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2[62185]: 2024-06-13T01:30:01.915+0000 7f8982583640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:30:02.212 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:01 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2[62185]: 2024-06-13T01:30:01.915+0000 7f8982583640 -1 osd.2 40 *** Got signal Terminated *** 2024-06-13T01:30:02.212 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:01 smithi132 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2[62185]: 2024-06-13T01:30:01.915+0000 7f8982583640 -1 osd.2 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T01:30:07.491 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:07 smithi132 podman[81083]: 2024-06-13 01:30:07.227407875 +0000 UTC m=+5.435171655 container died c662ef97b0fb4a56652521f8cc1182a82826944c2f1db708f455da8bb627110f (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-13T01:30:07.858 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:07 smithi132 podman[81083]: 2024-06-13 01:30:07.632507148 +0000 UTC m=+5.840270965 container cleanup c662ef97b0fb4a56652521f8cc1182a82826944c2f1db708f455da8bb627110f (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2, org.label-schema.build-date=20240612, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-13T01:30:07.858 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:07 smithi132 bash[81083]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2 2024-06-13T01:30:08.167 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:07 smithi132 podman[81109]: 2024-06-13 01:30:07.858878086 +0000 UTC m=+0.628278400 container remove c662ef97b0fb4a56652521f8cc1182a82826944c2f1db708f455da8bb627110f (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=quincy-296b94f, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2) 2024-06-13T01:30:08.462 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:08 smithi132 podman[81199]: 2024-06-13 01:30:08.168102118 +0000 UTC m=+0.103829070 container create a826c66e43caf05e19b7e92c56e617d457d4211a083be5a932de0d446d7cb8de (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, RELEASE=quincy-296b94f, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-13T01:30:08.462 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:08 smithi132 podman[81199]: 2024-06-13 01:30:08.081398802 +0000 UTC m=+0.017125758 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56 2024-06-13T01:30:08.462 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:08 smithi132 podman[81199]: 2024-06-13 01:30:08.387273682 +0000 UTC m=+0.323000628 container init a826c66e43caf05e19b7e92c56e617d457d4211a083be5a932de0d446d7cb8de (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_CLEAN=True, RELEASE=quincy-296b94f, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-13T01:30:08.462 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:08 smithi132 podman[81199]: 2024-06-13 01:30:08.390525858 +0000 UTC m=+0.326252805 container start a826c66e43caf05e19b7e92c56e617d457d4211a083be5a932de0d446d7cb8de (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2-deactivate, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-296b94f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-13T01:30:08.923 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:08 smithi132 podman[81199]: 2024-06-13 01:30:08.492014683 +0000 UTC m=+0.427741636 container attach a826c66e43caf05e19b7e92c56e617d457d4211a083be5a932de0d446d7cb8de (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=) 2024-06-13T01:30:08.924 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jun 13 01:30:08 smithi132 podman[81199]: 2024-06-13 01:30:08.631734437 +0000 UTC m=+0.567461386 container died a826c66e43caf05e19b7e92c56e617d457d4211a083be5a932de0d446d7cb8de (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-296b94f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T01:30:09.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.2.service' 2024-06-13T01:30:09.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:30:09.260 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-13T01:30:09.260 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-13T01:30:09.260 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.3 2024-06-13T01:30:09.598 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:09 smithi155 systemd[1]: Stopping Ceph osd.3 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:30:09.878 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:09 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3[41160]: 2024-06-13T01:30:09.598+0000 7f0d88ed9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:30:09.879 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:09 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3[41160]: 2024-06-13T01:30:09.598+0000 7f0d88ed9640 -1 osd.3 40 *** Got signal Terminated *** 2024-06-13T01:30:09.879 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:09 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3[41160]: 2024-06-13T01:30:09.598+0000 7f0d88ed9640 -1 osd.3 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T01:30:11.878 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:11 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:11.451+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:12.878 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:12 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:12.431+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:12.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:12 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:12.533+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:13.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:13 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:13.563+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:13.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:13 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:13.414+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:14.629 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:14 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:14.540+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:14.629 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:14 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3[41160]: 2024-06-13T01:30:14.340+0000 7f0d84cf0640 -1 osd.3 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:50.256755+0000 front 2024-06-13T01:29:50.256843+0000 (oldest deadline 2024-06-13T01:30:13.756655+0000) 2024-06-13T01:30:14.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:14 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:14.435+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:15.323 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:14 smithi155 podman[59625]: 2024-06-13 01:30:14.980588508 +0000 UTC m=+5.509359812 container died cb28a9019844725fcc283222afc81f214874307948b3cde5be2f2b5a148a8593 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-296b94f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-13T01:30:15.323 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:15 smithi155 podman[59625]: 2024-06-13 01:30:15.324176234 +0000 UTC m=+5.852947539 container cleanup cb28a9019844725fcc283222afc81f214874307948b3cde5be2f2b5a148a8593 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, GIT_CLEAN=True) 2024-06-13T01:30:15.628 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:15 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:15.530+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:15.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:15 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:15.442+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:15.629 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:15 smithi155 bash[59625]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3 2024-06-13T01:30:15.629 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:15 smithi155 podman[59640]: 2024-06-13 01:30:15.542473396 +0000 UTC m=+0.561333892 container remove cb28a9019844725fcc283222afc81f214874307948b3cde5be2f2b5a148a8593 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-296b94f, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-13T01:30:16.229 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:15 smithi155 podman[59733]: 2024-06-13 01:30:15.857993168 +0000 UTC m=+0.017422262 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56 2024-06-13T01:30:16.229 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:15 smithi155 podman[59733]: 2024-06-13 01:30:15.98572834 +0000 UTC m=+0.145157425 container create 9b3373865e48f3c97a3c3c2bfbe298ccb0561d780aad4b00296bd40f190293be (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-13T01:30:16.229 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:16 smithi155 podman[59733]: 2024-06-13 01:30:16.229485907 +0000 UTC m=+0.388914993 container init 9b3373865e48f3c97a3c3c2bfbe298ccb0561d780aad4b00296bd40f190293be (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T01:30:16.486 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:16 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:16.411+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:16.487 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:16 smithi155 podman[59733]: 2024-06-13 01:30:16.232784276 +0000 UTC m=+0.392213373 container start 9b3373865e48f3c97a3c3c2bfbe298ccb0561d780aad4b00296bd40f190293be (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=) 2024-06-13T01:30:16.487 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:16 smithi155 podman[59733]: 2024-06-13 01:30:16.321516558 +0000 UTC m=+0.480945652 container attach 9b3373865e48f3c97a3c3c2bfbe298ccb0561d780aad4b00296bd40f190293be (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-296b94f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-06-13T01:30:16.487 INFO:journalctl@ceph.osd.3.smithi155.stdout:Jun 13 01:30:16 smithi155 podman[59733]: 2024-06-13 01:30:16.477739711 +0000 UTC m=+0.637168798 container died 9b3373865e48f3c97a3c3c2bfbe298ccb0561d780aad4b00296bd40f190293be (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, RELEASE=quincy-296b94f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-13T01:30:16.823 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:16 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:16.486+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:17.152 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.3.service' 2024-06-13T01:30:17.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:30:17.192 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-13T01:30:17.192 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-13T01:30:17.193 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.4 2024-06-13T01:30:17.418 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:17 smithi155 systemd[1]: Stopping Ceph osd.4 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:30:17.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:17.418+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:17.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:17.454+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:17.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:17.536+0000 7f62da609640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:30:17.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:17.536+0000 7f62da609640 -1 osd.4 40 *** Got signal Terminated *** 2024-06-13T01:30:17.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:17 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:17.536+0000 7f62da609640 -1 osd.4 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T01:30:18.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:18 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:18.466+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:18.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:18 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:18.452+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:19.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:19 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:19.428+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:19.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:19 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:19.415+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:20.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:20 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:20.448+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:20.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:20 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:20.399+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:21.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:21 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:21.441+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:21.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:21 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:21.447+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:21.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:21 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:21.447+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:22.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:22 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:22.439+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:47.614688+0000 front 2024-06-13T01:29:47.614757+0000 (oldest deadline 2024-06-13T01:30:11.714670+0000) 2024-06-13T01:30:22.879 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:22 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4[47793]: 2024-06-13T01:30:22.439+0000 7f62d541e640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.915985+0000 front 2024-06-13T01:29:56.915970+0000 (oldest deadline 2024-06-13T01:30:22.215784+0000) 2024-06-13T01:30:22.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:22 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:22.435+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:22.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:22 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:22.435+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:23.220 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:22 smithi155 podman[59840]: 2024-06-13 01:30:22.956824469 +0000 UTC m=+5.542814571 container died dfc6e666b28366523e81fd6e0fb2b36d2d0f0c43aa992f3d76ed968fd4f5dc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-296b94f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-06-13T01:30:23.612 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:23 smithi155 podman[59840]: 2024-06-13 01:30:23.311332277 +0000 UTC m=+5.897322454 container cleanup dfc6e666b28366523e81fd6e0fb2b36d2d0f0c43aa992f3d76ed968fd4f5dc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-13T01:30:23.612 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:23 smithi155 bash[59840]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4 2024-06-13T01:30:23.612 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:23 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:23.439+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:23.613 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:23 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:23.439+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:23.878 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:23 smithi155 podman[59854]: 2024-06-13 01:30:23.612805767 +0000 UTC m=+0.653378181 container remove dfc6e666b28366523e81fd6e0fb2b36d2d0f0c43aa992f3d76ed968fd4f5dc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-13T01:30:24.274 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:23 smithi155 podman[59946]: 2024-06-13 01:30:23.839131019 +0000 UTC m=+0.017417804 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56 2024-06-13T01:30:24.274 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:23 smithi155 podman[59946]: 2024-06-13 01:30:23.946926604 +0000 UTC m=+0.125213432 container create 4f63dc429b40b54c49ce4670d5f2e84a418ae7bc23d8dbba378e1e53aee16fa6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-296b94f, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-13T01:30:24.274 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:24 smithi155 podman[59946]: 2024-06-13 01:30:24.182779348 +0000 UTC m=+0.361066130 container init 4f63dc429b40b54c49ce4670d5f2e84a418ae7bc23d8dbba378e1e53aee16fa6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-296b94f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-06-13T01:30:24.274 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:24 smithi155 podman[59946]: 2024-06-13 01:30:24.186124889 +0000 UTC m=+0.364411674 container start 4f63dc429b40b54c49ce4670d5f2e84a418ae7bc23d8dbba378e1e53aee16fa6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, RELEASE=quincy-296b94f, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-13T01:30:24.629 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:24 smithi155 podman[59946]: 2024-06-13 01:30:24.274366805 +0000 UTC m=+0.452653597 container attach 4f63dc429b40b54c49ce4670d5f2e84a418ae7bc23d8dbba378e1e53aee16fa6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-296b94f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T01:30:24.629 INFO:journalctl@ceph.osd.4.smithi155.stdout:Jun 13 01:30:24 smithi155 podman[59946]: 2024-06-13 01:30:24.44351167 +0000 UTC m=+0.621798455 container died 4f63dc429b40b54c49ce4670d5f2e84a418ae7bc23d8dbba378e1e53aee16fa6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240612, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-296b94f) 2024-06-13T01:30:24.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:24 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:24.391+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:24.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:24 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:24.391+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:25.096 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.4.service' 2024-06-13T01:30:25.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:30:25.135 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-13T01:30:25.135 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-13T01:30:25.135 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.5 2024-06-13T01:30:25.379 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:25 smithi155 systemd[1]: Stopping Ceph osd.5 for ad8d4854-2922-11ef-bc9c-c7b262605968... 2024-06-13T01:30:25.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:25 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:25.393+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:25.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:25 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:25.393+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:25.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:25 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:25.463+0000 7f63e8450640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-13T01:30:25.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:25 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:25.463+0000 7f63e8450640 -1 osd.5 40 *** Got signal Terminated *** 2024-06-13T01:30:25.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:25 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:25.463+0000 7f63e8450640 -1 osd.5 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-13T01:30:26.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:26 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:26.359+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:26.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:26 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:26.359+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:27.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:27 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:27.362+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:27.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:27 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:27.362+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:28.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:28 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:28.398+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:28.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:28 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:28.398+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:29.878 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:29 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:29.443+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:29.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:29 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:29.443+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:30.878 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:30 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:30.429+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6806 osd.0 since back 2024-06-13T01:29:46.038865+0000 front 2024-06-13T01:29:46.038806+0000 (oldest deadline 2024-06-13T01:30:11.338693+0000) 2024-06-13T01:30:30.879 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:30 smithi155 ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5[54196]: 2024-06-13T01:30:30.429+0000 7f63e1a62640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.132:6814 osd.1 since back 2024-06-13T01:29:56.539934+0000 front 2024-06-13T01:29:56.539883+0000 (oldest deadline 2024-06-13T01:30:21.140050+0000) 2024-06-13T01:30:31.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:30 smithi155 podman[60046]: 2024-06-13 01:30:30.954865616 +0000 UTC m=+5.598735328 container died 7951746cb095f931512a6abf4fc68d049f381110fa8b0e1a226419af73d19e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5, org.label-schema.schema-version=1.0, RELEASE=quincy-296b94f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, ceph=True) 2024-06-13T01:30:31.272 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:31 smithi155 podman[60046]: 2024-06-13 01:30:31.272359783 +0000 UTC m=+5.916229494 container cleanup 7951746cb095f931512a6abf4fc68d049f381110fa8b0e1a226419af73d19e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-296b94f, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-13T01:30:31.628 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:31 smithi155 bash[60046]: ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5 2024-06-13T01:30:31.629 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:31 smithi155 podman[60062]: 2024-06-13 01:30:31.490071469 +0000 UTC m=+0.533085941 container remove 7951746cb095f931512a6abf4fc68d049f381110fa8b0e1a226419af73d19e18 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-296b94f, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-06-13T01:30:32.167 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:31 smithi155 podman[60153]: 2024-06-13 01:30:31.882528789 +0000 UTC m=+0.092553601 container create d595c1ca1fcc04b365204a4c4a384761b91a4cca7e73a0b746cd690dd17910e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, RELEASE=quincy-296b94f, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-13T01:30:32.168 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:31 smithi155 podman[60153]: 2024-06-13 01:30:31.807500528 +0000 UTC m=+0.017525341 image pull 85997b5cbf2d76a1b949fbf8df1ec0cb1db8c7acb0bb2daecb752390ac6a830a quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56 2024-06-13T01:30:32.168 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:32 smithi155 podman[60153]: 2024-06-13 01:30:32.084902364 +0000 UTC m=+0.294927183 container init d595c1ca1fcc04b365204a4c4a384761b91a4cca7e73a0b746cd690dd17910e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-296b94f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-13T01:30:32.168 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:32 smithi155 podman[60153]: 2024-06-13 01:30:32.08823815 +0000 UTC m=+0.298262963 container start d595c1ca1fcc04b365204a4c4a384761b91a4cca7e73a0b746cd690dd17910e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, RELEASE=quincy-296b94f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-13T01:30:32.168 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:32 smithi155 podman[60153]: 2024-06-13 01:30:32.168018879 +0000 UTC m=+0.378043700 container attach d595c1ca1fcc04b365204a4c4a384761b91a4cca7e73a0b746cd690dd17910e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-296b94f, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, ceph=True, GIT_CLEAN=True) 2024-06-13T01:30:32.603 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:32 smithi155 podman[60153]: 2024-06-13 01:30:32.337326532 +0000 UTC m=+0.547351345 container died d595c1ca1fcc04b365204a4c4a384761b91a4cca7e73a0b746cd690dd17910e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, RELEASE=quincy-296b94f, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-13T01:30:32.854 INFO:journalctl@ceph.osd.5.smithi155.stdout:Jun 13 01:30:32 smithi155 podman[60177]: 2024-06-13 01:30:32.837903538 +0000 UTC m=+0.494220436 container remove d595c1ca1fcc04b365204a4c4a384761b91a4cca7e73a0b746cd690dd17910e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ff2324942a43cf7e8825188f314727394da12489e8056d8d2171cb306b040c56, name=ceph-ad8d4854-2922-11ef-bc9c-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-296b94f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0) 2024-06-13T01:30:32.857 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ad8d4854-2922-11ef-bc9c-c7b262605968@osd.5.service' 2024-06-13T01:30:32.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-13T01:30:32.895 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-13T01:30:32.895 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 --force --keep-logs 2024-06-13T01:30:41.922 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 --force --keep-logs 2024-06-13T01:30:47.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:30:47.176 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-13T01:30:47.203 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-13T01:30:47.205 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/remote/smithi132/crash 2024-06-13T01:30:47.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/crash -- . 2024-06-13T01:30:47.271 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/crash: Cannot open: No such file or directory 2024-06-13T01:30:47.271 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-06-13T01:30:47.273 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/remote/smithi155/crash 2024-06-13T01:30:47.274 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/crash -- . 2024-06-13T01:30:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/crash: Cannot open: No such file or directory 2024-06-13T01:30:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2024-06-13T01:30:47.312 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-13T01:30:47.312 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-13T01:30:47.346 INFO:teuthology.orchestra.run.smithi132.stdout:1718241579.5358806 mon.smithi132 (mon.0) 119 : cluster [WRN] Health check failed: 2 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-13T01:30:47.346 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-13T01:30:47.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[SEC\]' /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-13T01:30:47.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]' /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-13T01:30:47.490 INFO:teuthology.orchestra.run.smithi132.stdout:1718241643.8654923 mgr.smithi155.teuqcw (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents 2024-06-13T01:30:47.491 INFO:tasks.cephadm:Compressing logs... 2024-06-13T01:30:47.492 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-13T01:30:47.534 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-13T01:30:47.557 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log 2024-06-13T01:30:47.557 INFO:teuthology.orchestra.run.smithi132.stderr:: No such file or directory 2024-06-13T01:30:47.558 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.2.log 2024-06-13T01:30:47.558 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.audit.log 2024-06-13T01:30:47.558 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log: /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.1.log 2024-06-13T01:30:47.558 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.0.log 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mon.smithi132.log 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr: 87.4% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log.gz 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.1.log: /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mgr.smithi132.vjweoa.log 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mon.smithi132.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.cephadm.log 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mgr.smithi132.vjweoa.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-volume.log 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr: 90.1% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.audit.log.gz 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-13T01:30:47.563 INFO:teuthology.orchestra.run.smithi132.stderr: 83.5% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.cephadm.log.gz/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-volume.log: 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr:: No such file or directory 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mgr.smithi155.teuqcw.log 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.audit.log 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log: /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mgr.smithi155.teuqcw.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.5.log 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.3.log 2024-06-13T01:30:47.564 INFO:teuthology.orchestra.run.smithi155.stderr: 87.7% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.log.gz 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.4.log 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.5.log: /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.cephadm.log 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mon.smithi155.log 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr: 90.3% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.audit.log.gz 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-volume.log 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr: 82.0% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph.cephadm.log.gz 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose --/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mon.smithi155.log: /var/log/ceph/cephadm.log 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-13T01:30:47.565 INFO:teuthology.orchestra.run.smithi155.stderr: 92.5% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mgr.smithi155.teuqcw.log.gz 2024-06-13T01:30:47.568 INFO:teuthology.orchestra.run.smithi155.stderr: 90.9% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-volume.log.gz 2024-06-13T01:30:47.570 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: 93.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-13T01:30:47.571 INFO:teuthology.orchestra.run.smithi132.stderr: 90.9% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-volume.log.gz 2024-06-13T01:30:47.603 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.5.log.gz 2024-06-13T01:30:47.604 INFO:teuthology.orchestra.run.smithi155.stderr: 92.9% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.4.log.gz 2024-06-13T01:30:47.617 INFO:teuthology.orchestra.run.smithi132.stderr: 93.1% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.2.log.gz 2024-06-13T01:30:47.626 INFO:teuthology.orchestra.run.smithi155.stderr: 92.5% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.3.log.gz 2024-06-13T01:30:47.645 INFO:teuthology.orchestra.run.smithi132.stderr: 93.3% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.1.log.gz 2024-06-13T01:30:47.650 INFO:teuthology.orchestra.run.smithi132.stderr: 92.7% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-osd.0.log.gz 2024-06-13T01:30:47.654 INFO:teuthology.orchestra.run.smithi155.stderr: 92.8% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mon.smithi155.log.gz 2024-06-13T01:30:47.656 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-06-13T01:30:47.656 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m0.107s 2024-06-13T01:30:47.656 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m0.257s 2024-06-13T01:30:47.656 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.034s 2024-06-13T01:30:47.658 INFO:teuthology.orchestra.run.smithi132.stderr: 89.2% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mgr.smithi132.vjweoa.log.gz 2024-06-13T01:30:47.849 INFO:teuthology.orchestra.run.smithi132.stderr: 90.5% -- replaced with /var/log/ceph/ad8d4854-2922-11ef-bc9c-c7b262605968/ceph-mon.smithi132.log.gz 2024-06-13T01:30:47.850 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-06-13T01:30:47.851 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m0.303s 2024-06-13T01:30:47.851 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m0.637s 2024-06-13T01:30:47.851 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m0.039s 2024-06-13T01:30:47.851 INFO:tasks.cephadm:Archiving logs... 2024-06-13T01:30:47.851 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/remote/smithi132/log 2024-06-13T01:30:47.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-13T01:30:48.017 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/remote/smithi155/log 2024-06-13T01:30:48.018 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-13T01:30:48.090 INFO:tasks.cephadm:Removing cluster... 2024-06-13T01:30:48.090 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 --force 2024-06-13T01:30:48.559 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ad8d4854-2922-11ef-bc9c-c7b262605968 --force 2024-06-13T01:30:49.026 INFO:tasks.cephadm:Removing cephadm ... 2024-06-13T01:30:49.027 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-13T01:30:49.043 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-13T01:30:49.058 INFO:tasks.cephadm:Teardown complete 2024-06-13T01:30:49.059 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-13T01:30:49.232 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e9bbe1d025f643daadd449db7dfa71c9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_1373963c202323287783ccb099bbae1ec8f14a5d/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-13T01:30:49.234 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-13T01:30:49.244 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-13T01:30:49.257 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-13T01:30:49.257 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-13T01:30:49.269 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-13T01:30:49.336 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-13T01:30:49.345 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-13T01:30:49.345 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T01:30:49.348 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-13T01:30:49.360 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-06-13T01:30:49.362 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-06-13T01:30:49.364 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T01:30:49.365 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-06-13T01:30:49.365 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 +793us[ +808us] +/- 45ms 2024-06-13T01:30:49.365 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 42 -1347us[-1331us] +/- 61ms 2024-06-13T01:30:49.365 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +227us[ +243us] +/- 62ms 2024-06-13T01:30:49.365 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:30:49.366 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-13T01:30:49.367 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-06-13T01:30:49.367 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +815us[ +854us] +/- 45ms 2024-06-13T01:30:49.367 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -1325us[-1325us] +/- 61ms 2024-06-13T01:30:49.367 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +264us[ +264us] +/- 62ms 2024-06-13T01:30:49.367 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-13T01:30:49.367 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-13T01:30:49.378 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-13T01:30:49.379 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-13T01:30:49.393 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-13T01:30:49.475 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-13T01:30:49.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T01:30:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:30:49.566 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-13T01:30:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:30:49.594 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-13T01:30:49.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-13T01:30:49.613 INFO:teuthology.task.internal:Duration was 1212.106140 seconds 2024-06-13T01:30:49.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-13T01:30:49.623 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-13T01:30:49.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-13T01:30:49.626 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-13T01:30:49.661 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T01:30:49.674 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-13T01:30:50.006 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-13T01:30:50.006 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-06-13T01:30:50.006 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-13T01:30:50.066 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-06-13T01:30:50.066 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-13T01:30:50.089 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-13T01:30:50.090 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-13T01:30:50.109 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-13T01:30:50.156 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-13T01:30:50.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-13T01:30:50.177 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-13T01:30:50.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-13T01:30:50.568 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-13T01:30:50.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-13T01:30:50.593 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-13T01:30:50.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-13T01:30:50.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-13T01:30:50.691 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-13T01:30:50.714 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-06-13T01:30:50.714 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-06-13T01:30:50.753 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-13T01:30:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:30:50.801 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-13T01:30:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-13T01:30:50.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-13T01:30:50.829 INFO:teuthology.task.internal:Transferring archived files... 2024-06-13T01:30:50.830 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/remote/smithi132 2024-06-13T01:30:50.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-13T01:30:50.896 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-12_21:24:03-fs-squid-distro-default-smithi/7752577/remote/smithi155 2024-06-13T01:30:50.896 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-13T01:30:50.936 INFO:teuthology.task.internal:Removing archive directory... 2024-06-13T01:30:50.936 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-13T01:30:50.941 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-13T01:30:50.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-13T01:30:50.998 INFO:teuthology.task.internal:Not uploading archives. 2024-06-13T01:30:50.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-13T01:30:51.015 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-13T01:30:51.015 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-13T01:30:51.018 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-13T01:30:51.030 INFO:teuthology.orchestra.run.smithi132.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 13 01:30 /home/ubuntu/cephtest 2024-06-13T01:30:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 13 01:30 /home/ubuntu/cephtest 2024-06-13T01:30:51.042 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-13T01:30:51.053 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-13T01:30:51.148 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1212.1061401367188 failure_reason: '"1718241643.8654923 mgr.smithi155.teuqcw (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e9bbe1d025f643daadd449db7dfa71c9 status: fail success: false 2024-06-13T01:30:51.149 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-13T01:30:51.232 INFO:teuthology.run:FAIL