2024-05-06T14:12:42.262 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-06T14:12:42.262 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254 --owner scheduled_teuthology@teuthology --name teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/orig.config.yaml 2024-05-06T14:12:42.320 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T14:12:42.411 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7688254' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.0 ' name: teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3735 sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNDhQqShhO5NXWOND1UifgILZQ1beRwhLV9L73gNkq1ZKTMDxl1bv6nRVSwT83u386wiZyUy7YCMTPc1oy2BWM= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6o9s/jwV00BMmczw71GWwoGiRRXXko1PMNfztbEN9AuXPcG6uVIS6734lMITpMwqZj7tL/2ZowTGWJ7f6WL/M= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.0 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.0 roleless: true - print: '**** done end installing v18.2.0 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-05-06T14:12:42.411 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa; will attempt to use it 2024-05-06T14:12:42.412 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks 2024-05-06T14:12:42.412 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-06T14:12:42.414 INFO:teuthology.task.internal:Checking packages... 2024-05-06T14:12:42.439 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '70ce9539b62a15b9de429c1f0e91a1e012b099aa' 2024-05-06T14:12:42.439 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-06T14:12:42.440 INFO:teuthology.packaging:ref: None 2024-05-06T14:12:42.440 INFO:teuthology.packaging:tag: None 2024-05-06T14:12:42.440 INFO:teuthology.packaging:branch: squid 2024-05-06T14:12:42.440 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:12:42.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-06T14:12:42.839 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2504.gb05eb275 2024-05-06T14:12:42.841 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-06T14:12:42.847 INFO:teuthology.task.internal:no buildpackages task found 2024-05-06T14:12:42.847 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-06T14:12:42.853 INFO:teuthology.task.internal:Saving configuration 2024-05-06T14:12:42.874 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-06T14:12:42.880 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T14:12:42.905 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 14:07:30.112023', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNDhQqShhO5NXWOND1UifgILZQ1beRwhLV9L73gNkq1ZKTMDxl1bv6nRVSwT83u386wiZyUy7YCMTPc1oy2BWM='} 2024-05-06T14:12:42.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 14:07:30.110001', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6o9s/jwV00BMmczw71GWwoGiRRXXko1PMNfztbEN9AuXPcG6uVIS6734lMITpMwqZj7tL/2ZowTGWJ7f6WL/M='} 2024-05-06T14:12:42.926 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-06T14:12:43.003 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-05-06T14:12:43.004 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-05-06T14:12:43.004 INFO:teuthology.run_tasks:Running task console_log... 2024-05-06T14:12:43.162 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7485e9a0d0>, signals=[15]) 2024-05-06T14:12:43.163 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-06T14:12:43.169 INFO:teuthology.task.internal:Opening connections... 2024-05-06T14:12:43.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:12:43.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T14:12:43.249 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-05-06T14:12:43.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T14:12:43.324 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-06T14:12:43.331 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-05-06T14:12:43.349 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-05-06T14:12:43.349 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-05-06T14:12:43.406 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T14:12:43.407 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T14:12:43.408 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-05-06T14:12:43.428 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-05-06T14:12:43.445 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-05-06T14:12:43.446 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-05-06T14:12:43.503 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-05-06T14:12:43.503 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-05-06T14:12:43.503 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-05-06T14:12:43.503 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-05-06T14:12:43.503 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T14:12:43.504 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T14:12:43.505 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-05-06T14:12:43.527 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-06T14:12:43.537 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-06T14:12:43.542 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-06T14:12:43.542 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-05-06T14:12:43.545 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-05-06T14:12:43.561 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-06T14:12:43.566 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-06T14:12:43.567 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-05-06T14:12:43.603 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-05-06T14:12:43.626 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-06T14:12:43.685 INFO:teuthology.run_tasks:Running task kernel... 2024-05-06T14:12:43.706 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-06T14:12:43.706 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-06T14:12:43.706 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T14:12:43.707 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T14:12:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:12:43.724 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-05-06T14:12:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:12:43.725 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-05-06T14:12:43.781 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-437.el9.x86_64 2024-05-06T14:12:43.781 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-437.el9.x86_64 2024-05-06T14:12:43.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-05-06T14:12:43.783 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-437.el9.x86_64 2024-05-06T14:12:43.783 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-437.el9.x86_64 2024-05-06T14:12:43.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-05-06T14:12:46.290 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.0 MB 00:00 2024-05-06T14:12:46.764 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-05-06T14:12:48.157 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-06T14:12:48.448 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-05-06T14:12:52.582 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-06T14:12:53.324 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.2 MB 00:01 2024-05-06T14:12:54.049 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-05-06T14:12:54.882 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 96 kB/s | 16 kB 00:00 2024-05-06T14:12:56.262 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-05-06T14:13:01.417 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-06T14:13:03.324 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T14:13:03.324 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T14:13:03.416 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:13:03.417 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:13:03.417 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:13:03.545 DEBUG:teuthology.orchestra.run.smithi190:> echo no | sudo yum reinstall kernel || true 2024-05-06T14:13:04.046 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 02:13:01 PM UTC. 2024-05-06T14:13:04.179 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:13:04.189 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout:Reinstalling: 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 255 k 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 0 2024-05-06T14:13:04.190 INFO:teuthology.orchestra.run.smithi190.stderr:Operation aborted. 2024-05-06T14:13:04.268 INFO:teuthology.orchestra.run.smithi190.stdout:Is this ok [y/N]: 2024-05-06T14:13:04.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum reinstall -y kernel || true 2024-05-06T14:13:04.766 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 02:13:01 PM UTC. 2024-05-06T14:13:04.882 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout:Reinstalling: 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:13:04.883 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:13:04.884 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:13:04.884 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:13:04.884 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 255 k 2024-05-06T14:13:04.884 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 0 2024-05-06T14:13:04.884 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:13:05.594 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-437.el9.x86_64.rpm 431 kB/s | 255 kB 00:00 2024-05-06T14:13:05.594 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:13:05.594 INFO:teuthology.orchestra.run.smithi190.stdout:Total 359 kB/s | 255 kB 00:00 2024-05-06T14:13:05.614 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:13:05.807 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:13:05.808 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:13:05.813 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:13:05.813 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:13:05.935 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:13:06.011 INFO:teuthology.orchestra.run.smithi190.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T14:13:06.012 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T14:13:06.671 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T14:13:06.671 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T14:13:07.385 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T14:13:07.385 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:13:07.385 INFO:teuthology.orchestra.run.smithi190.stdout:Reinstalled: 2024-05-06T14:13:07.386 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T14:13:07.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:13:07.386 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:13:07.564 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T14:13:07.651 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T14:13:07.651 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T14:13:07.651 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T14:13:07.651 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-05-06T14:13:07.652 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T14:13:07.687 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-05-06T14:13:07.709 DEBUG:teuthology.parallel:result is None 2024-05-06T14:15:05.767 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 165 kB/s | 21 MB 02:10 2024-05-06T14:15:10.914 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-05-06T14:15:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:01 ago on Mon 06 May 2024 02:15:10 PM UTC. 2024-05-06T14:15:12.803 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T14:15:12.803 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T14:15:12.890 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:15:12.891 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:15:12.892 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:15:12.992 DEBUG:teuthology.orchestra.run.smithi153:> echo no | sudo yum reinstall kernel || true 2024-05-06T14:15:13.494 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 02:15:10 PM UTC. 2024-05-06T14:15:13.621 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout:Reinstalling: 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:15:13.638 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:15:13.639 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 255 k 2024-05-06T14:15:13.639 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 0 2024-05-06T14:15:13.639 INFO:teuthology.orchestra.run.smithi153.stderr:Operation aborted. 2024-05-06T14:15:13.692 INFO:teuthology.orchestra.run.smithi153.stdout:Is this ok [y/N]: 2024-05-06T14:15:13.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum reinstall -y kernel || true 2024-05-06T14:15:14.186 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:04 ago on Mon 06 May 2024 02:15:10 PM UTC. 2024-05-06T14:15:14.302 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout:Reinstalling: 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:15:14.303 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:15:14.304 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:15:14.304 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:15:14.304 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 255 k 2024-05-06T14:15:14.304 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 0 2024-05-06T14:15:14.304 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:15:15.166 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-437.el9.x86_64.rpm 344 kB/s | 255 kB 00:00 2024-05-06T14:15:15.166 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:15:15.166 INFO:teuthology.orchestra.run.smithi153.stdout:Total 295 kB/s | 255 kB 00:00 2024-05-06T14:15:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:15:15.381 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:15:15.381 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:15:15.386 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:15:15.386 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:15:15.533 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:15:15.625 INFO:teuthology.orchestra.run.smithi153.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T14:15:15.625 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T14:15:16.114 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T14:15:16.114 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T14:15:16.837 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T14:15:16.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:15:16.837 INFO:teuthology.orchestra.run.smithi153.stdout:Reinstalled: 2024-05-06T14:15:16.837 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T14:15:16.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:15:16.837 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:15:17.013 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T14:15:17.100 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T14:15:17.101 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T14:15:17.101 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T14:15:17.101 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-05-06T14:15:17.101 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T14:15:17.130 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-05-06T14:15:17.154 DEBUG:teuthology.parallel:result is None 2024-05-06T14:15:17.155 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-06T14:15:17.164 INFO:teuthology.task.internal:Creating test directory... 2024-05-06T14:15:17.164 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T14:15:17.197 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T14:15:17.215 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-06T14:15:17.221 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-06T14:15:17.228 INFO:teuthology.task.internal:Creating archive directory... 2024-05-06T14:15:17.228 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T14:15:17.253 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T14:15:17.282 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-06T14:15:17.291 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-06T14:15:17.291 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T14:15:17.317 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T14:15:17.346 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T14:15:17.356 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T14:15:17.390 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T14:15:17.401 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T14:15:17.402 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-06T14:15:17.410 INFO:teuthology.task.internal:Configuring sudo... 2024-05-06T14:15:17.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T14:15:17.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T14:15:17.438 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-06T14:15:17.448 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-06T14:15:17.448 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T14:15:17.477 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T14:15:17.495 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T14:15:17.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T14:15:17.615 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T14:15:17.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T14:15:17.736 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:15:17.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T14:15:17.799 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T14:15:17.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T14:15:17.893 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T14:15:17.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T14:15:18.018 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:15:18.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T14:15:18.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-05-06T14:15:18.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-05-06T14:15:18.112 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T14:15:18.166 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T14:15:18.572 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-06T14:15:18.580 INFO:teuthology.task.internal:Starting timer... 2024-05-06T14:15:18.580 INFO:teuthology.run_tasks:Running task pcp... 2024-05-06T14:15:18.590 INFO:teuthology.run_tasks:Running task selinux... 2024-05-06T14:15:18.599 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-06T14:15:18.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-05-06T14:15:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-05-06T14:15:18.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-05-06T14:15:18.711 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-05-06T14:15:18.712 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-06T14:15:18.713 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-05-06T14:15:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-05-06T14:15:18.737 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-05-06T14:15:18.760 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-05-06T14:15:18.760 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-05-06T14:15:18.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-06T14:15:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:15:18.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-06T14:15:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:15:18.827 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-06T14:15:18.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-05-06T14:15:18.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-05-06T14:15:18.890 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-06T14:15:18.900 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-06T14:15:18.906 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-06T14:15:19.038 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-06T14:15:19.054 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-06T14:15:19.055 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-06T14:19:17.121 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-05-06T14:19:17.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-05-06T14:19:17.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T14:19:17.202 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-05-06T14:19:17.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-05-06T14:19:17.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-05-06T14:19:17.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T14:19:17.373 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-05-06T14:19:17.461 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-05-06T14:19:17.462 INFO:teuthology.run_tasks:Running task clock... 2024-05-06T14:19:17.473 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-06T14:19:17.473 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T14:19:17.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T14:19:17.476 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T14:19:17.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T14:19:17.503 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T14:19:17.517 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T14:19:17.528 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T14:19:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-05-06T14:19:17.541 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T14:19:17.550 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-05-06T14:19:17.563 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T14:19:17.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-05-06T14:19:17.573 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-05-06T14:19:17.576 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T14:19:17.586 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T14:19:17.599 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T14:19:17.632 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-05-06T14:19:17.636 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T14:19:17.637 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-05-06T14:19:17.637 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.637 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.637 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.637 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.655 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-05-06T14:19:17.658 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T14:19:17.658 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-05-06T14:19:17.659 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.659 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.659 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.659 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:19:17.660 INFO:teuthology.run_tasks:Running task install... 2024-05-06T14:19:17.669 DEBUG:teuthology.task.install:project ceph 2024-05-06T14:19:17.669 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa'}} 2024-05-06T14:19:17.669 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.0', 'flavor': 'default', 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa'} 2024-05-06T14:19:17.669 INFO:teuthology.task.install:Using flavor: default 2024-05-06T14:19:17.676 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-06T14:19:17.676 INFO:teuthology.task.install:extra packages: [] 2024-05-06T14:19:17.677 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-05-06T14:19:17.677 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-06T14:19:17.677 INFO:teuthology.packaging:ref: None 2024-05-06T14:19:17.677 INFO:teuthology.packaging:tag: v18.2.0 2024-05-06T14:19:17.678 INFO:teuthology.packaging:branch: None 2024-05-06T14:19:17.678 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:19:20.604 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.0^{} -> 5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-05-06T14:19:20.605 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-05-06T14:19:20.607 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa', 'tag': 'v18.2.0', 'wait_for_package': False} 2024-05-06T14:19:20.608 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-06T14:19:20.608 INFO:teuthology.packaging:ref: None 2024-05-06T14:19:20.608 INFO:teuthology.packaging:tag: v18.2.0 2024-05-06T14:19:20.608 INFO:teuthology.packaging:branch: None 2024-05-06T14:19:20.608 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:19:20.608 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-05-06T14:19:20.751 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-05-06T14:19:20.751 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-05-06T14:19:20.763 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/ 2024-05-06T14:19:20.763 INFO:teuthology.task.install.rpm:Package version is 18.2.0-0 2024-05-06T14:19:20.791 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-06T14:19:20.791 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:19:20.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-06T14:19:20.799 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/5dd24139a1eada541a3bc16b6941c5dde975e26d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-06T14:19:20.799 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:19:20.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-06T14:19:20.821 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-06T14:19:20.822 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-06T14:19:20.822 INFO:teuthology.packaging:ref: None 2024-05-06T14:19:20.822 INFO:teuthology.packaging:tag: v18.2.0 2024-05-06T14:19:20.822 INFO:teuthology.packaging:branch: None 2024-05-06T14:19:20.822 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:19:20.822 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-06T14:19:20.829 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-06T14:19:20.830 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-06T14:19:20.830 INFO:teuthology.packaging:ref: None 2024-05-06T14:19:20.830 INFO:teuthology.packaging:tag: v18.2.0 2024-05-06T14:19:20.830 INFO:teuthology.packaging:branch: None 2024-05-06T14:19:20.830 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:19:20.830 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-06T14:19:20.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-06T14:19:20.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-06T14:19:20.965 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-06T14:19:20.976 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-06T14:19:21.029 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-05-06T14:19:21.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-05-06T14:19:21.041 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-05-06T14:19:21.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-05-06T14:19:21.260 INFO:teuthology.orchestra.run.smithi153.stdout:50 files removed 2024-05-06T14:19:21.271 INFO:teuthology.orchestra.run.smithi190.stdout:50 files removed 2024-05-06T14:19:21.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-05-06T14:19:21.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-05-06T14:19:21.728 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 419 kB/s | 76 kB 00:00 2024-05-06T14:19:21.763 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 370 kB/s | 76 kB 00:00 2024-05-06T14:19:21.953 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 51 kB/s | 9.3 kB 00:00 2024-05-06T14:19:21.963 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 60 kB/s | 9.3 kB 00:00 2024-05-06T14:19:22.153 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 13 kB/s | 2.2 kB 00:00 2024-05-06T14:19:22.172 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 12 kB/s | 2.2 kB 00:00 2024-05-06T14:19:24.864 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-05-06T14:19:26.120 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-05-06T14:19:26.940 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-05-06T14:19:27.904 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-05-06T14:19:31.032 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-05-06T14:19:33.650 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-05-06T14:19:33.704 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.2 MB 00:02 2024-05-06T14:19:35.239 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-05-06T14:19:36.974 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-06T14:19:38.796 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 21 MB 00:05 2024-05-06T14:19:42.190 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-06T14:19:44.041 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-06T14:19:44.315 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:19:44.317 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:19:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-05-06T14:19:44.317 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:19:44.318 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:19:44.318 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-05-06T14:19:44.318 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-05-06T14:19:44.318 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-05-06T14:19:44.318 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-05-06T14:19:44.319 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:19:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-06T14:19:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-05-06T14:19:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-05-06T14:19:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-05-06T14:19:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-05-06T14:19:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-06T14:19:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-06T14:19:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-06T14:19:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-05-06T14:19:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-06T14:19:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-06T14:19:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-06T14:19:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-05-06T14:19:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-06T14:19:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-05-06T14:19:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-06T14:19:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-06T14:19:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-06T14:19:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-05-06T14:19:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-05-06T14:19:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-05-06T14:19:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-06T14:19:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-05-06T14:19:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-05-06T14:19:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-05-06T14:19:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-06T14:19:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-06T14:19:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:19:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:19:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:19:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:Install 28 Packages 2024-05-06T14:19:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-05-06T14:19:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:19:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 53 M 2024-05-06T14:19:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:19:44.775 INFO:teuthology.orchestra.run.smithi153.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 27 MB/s | 5.2 MB 00:00 2024-05-06T14:19:44.800 INFO:teuthology.orchestra.run.smithi153.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 975 kB/s | 24 kB 00:00 2024-05-06T14:19:44.834 INFO:teuthology.orchestra.run.smithi153.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 19 MB/s | 653 kB 00:00 2024-05-06T14:19:44.867 INFO:teuthology.orchestra.run.smithi153.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-05-06T14:19:44.984 INFO:teuthology.orchestra.run.smithi153.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 19 MB/s | 7.6 MB 00:00 2024-05-06T14:19:45.009 INFO:teuthology.orchestra.run.smithi153.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-05-06T14:19:45.034 INFO:teuthology.orchestra.run.smithi153.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.7 MB/s | 119 kB 00:00 2024-05-06T14:19:45.060 INFO:teuthology.orchestra.run.smithi153.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 155 kB 00:00 2024-05-06T14:19:45.119 INFO:teuthology.orchestra.run.smithi153.stdout:(9/30): librgw2-18.2.0-0.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-06T14:19:45.144 INFO:teuthology.orchestra.run.smithi153.stdout:(10/30): python3-rados-18.2.0-0.el9.x86_64.rpm 3.8 MB/s | 321 kB 00:00 2024-05-06T14:19:45.169 INFO:teuthology.orchestra.run.smithi153.stdout:(11/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 5.8 MB/s | 297 kB 00:00 2024-05-06T14:19:45.194 INFO:teuthology.orchestra.run.smithi153.stdout:(12/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-05-06T14:19:45.403 INFO:teuthology.orchestra.run.smithi153.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-05-06T14:19:45.470 INFO:teuthology.orchestra.run.smithi153.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-05-06T14:19:45.520 INFO:teuthology.orchestra.run.smithi153.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-05-06T14:19:45.546 INFO:teuthology.orchestra.run.smithi153.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 457 kB/s | 160 kB 00:00 2024-05-06T14:19:45.596 INFO:teuthology.orchestra.run.smithi153.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-05-06T14:19:45.638 INFO:teuthology.orchestra.run.smithi153.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-05-06T14:19:45.705 INFO:teuthology.orchestra.run.smithi153.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-05-06T14:19:45.796 INFO:teuthology.orchestra.run.smithi153.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 552 kB/s | 111 kB 00:00 2024-05-06T14:19:45.863 INFO:teuthology.orchestra.run.smithi153.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-05-06T14:19:45.891 INFO:teuthology.orchestra.run.smithi153.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-05-06T14:19:45.922 INFO:teuthology.orchestra.run.smithi153.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-05-06T14:19:45.939 INFO:teuthology.orchestra.run.smithi153.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-06T14:19:46.048 INFO:teuthology.orchestra.run.smithi153.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-05-06T14:19:46.193 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:19:46.195 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:18.2.0-0.el9 ceph 7.6 M 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:18.2.0-0.el9 ceph 3.3 M 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:18.2.0-0.el9 ceph 5.2 M 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:18.2.0-0.el9 ceph 18 M 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:18.2.0-0.el9 ceph 24 k 2024-05-06T14:19:46.196 INFO:teuthology.orchestra.run.smithi190.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:18.2.0-0.el9 ceph 653 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:18.2.0-0.el9 ceph 474 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-06T14:19:46.197 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:18.2.0-0.el9 ceph 45 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:18.2.0-0.el9 ceph 119 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:18.2.0-0.el9 ceph 155 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:18.2.0-0.el9 ceph 321 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:18.2.0-0.el9 ceph 297 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:18.2.0-0.el9 ceph 99 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout:Install 28 Packages 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-05-06T14:19:46.198 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:19:46.199 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 53 M 2024-05-06T14:19:46.199 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:19:46.215 INFO:teuthology.orchestra.run.smithi153.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.8 MB/s | 839 kB 00:00 2024-05-06T14:19:46.273 INFO:teuthology.orchestra.run.smithi153.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-05-06T14:19:46.315 INFO:teuthology.orchestra.run.smithi153.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-05-06T14:19:46.390 INFO:teuthology.orchestra.run.smithi153.stdout:(29/30): librados2-18.2.0-0.el9.x86_64.rpm 19 MB/s | 3.3 MB 00:00 2024-05-06T14:19:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:(30/30): librbd1-18.2.0-0.el9.x86_64.rpm 18 MB/s | 3.0 MB 00:00 2024-05-06T14:19:46.443 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:19:46.443 INFO:teuthology.orchestra.run.smithi153.stdout:Total 25 MB/s | 53 MB 00:02 2024-05-06T14:19:46.513 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:19:46.549 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:19:46.549 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:19:46.835 INFO:teuthology.orchestra.run.smithi190.stdout:(1/30): ceph-base-18.2.0-0.el9.x86_64.rpm 27 MB/s | 5.2 MB 00:00 2024-05-06T14:19:46.860 INFO:teuthology.orchestra.run.smithi190.stdout:(2/30): ceph-selinux-18.2.0-0.el9.x86_64.rpm 975 kB/s | 24 kB 00:00 2024-05-06T14:19:46.902 INFO:teuthology.orchestra.run.smithi190.stdout:(3/30): libcephfs2-18.2.0-0.el9.x86_64.rpm 15 MB/s | 653 kB 00:00 2024-05-06T14:19:46.936 INFO:teuthology.orchestra.run.smithi190.stdout:(4/30): libradosstriper1-18.2.0-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-05-06T14:19:46.967 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:19:46.967 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:19:47.044 INFO:teuthology.orchestra.run.smithi190.stdout:(5/30): ceph-radosgw-18.2.0-0.el9.x86_64.rpm 19 MB/s | 7.6 MB 00:00 2024-05-06T14:19:47.069 INFO:teuthology.orchestra.run.smithi190.stdout:(6/30): python3-ceph-argparse-18.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-05-06T14:19:47.095 INFO:teuthology.orchestra.run.smithi190.stdout:(7/30): python3-ceph-common-18.2.0-0.el9.x86_64 4.7 MB/s | 119 kB 00:00 2024-05-06T14:19:47.120 INFO:teuthology.orchestra.run.smithi190.stdout:(8/30): python3-cephfs-18.2.0-0.el9.x86_64.rpm 6.1 MB/s | 155 kB 00:00 2024-05-06T14:19:47.145 INFO:teuthology.orchestra.run.smithi190.stdout:(9/30): python3-rados-18.2.0-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-05-06T14:19:47.171 INFO:teuthology.orchestra.run.smithi190.stdout:(10/30): python3-rbd-18.2.0-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-05-06T14:19:47.196 INFO:teuthology.orchestra.run.smithi190.stdout:(11/30): python3-rgw-18.2.0-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-05-06T14:19:47.255 INFO:teuthology.orchestra.run.smithi190.stdout:(12/30): librgw2-18.2.0-0.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-06T14:19:47.447 INFO:teuthology.orchestra.run.smithi190.stdout:(13/30): ceph-common-18.2.0-0.el9.x86_64.rpm 23 MB/s | 18 MB 00:00 2024-05-06T14:19:47.597 INFO:teuthology.orchestra.run.smithi190.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 267 kB/s | 107 kB 00:00 2024-05-06T14:19:47.622 INFO:teuthology.orchestra.run.smithi190.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 259 kB/s | 45 kB 00:00 2024-05-06T14:19:47.648 INFO:teuthology.orchestra.run.smithi190.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 408 kB/s | 160 kB 00:00 2024-05-06T14:19:47.706 INFO:teuthology.orchestra.run.smithi190.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 727 kB/s | 42 kB 00:00 2024-05-06T14:19:47.740 INFO:teuthology.orchestra.run.smithi190.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-05-06T14:19:47.773 INFO:teuthology.orchestra.run.smithi190.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-05-06T14:19:48.107 INFO:teuthology.orchestra.run.smithi190.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 276 kB/s | 111 kB 00:00 2024-05-06T14:19:48.132 INFO:teuthology.orchestra.run.smithi190.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 787 kB/s | 309 kB 00:00 2024-05-06T14:19:48.233 INFO:teuthology.orchestra.run.smithi190.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-05-06T14:19:48.349 INFO:teuthology.orchestra.run.smithi190.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 224 kB/s | 49 kB 00:00 2024-05-06T14:19:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:19:48.376 INFO:teuthology.orchestra.run.smithi190.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2024-05-06T14:19:48.458 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-05-06T14:19:48.487 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-05-06T14:19:48.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-05-06T14:19:48.617 INFO:teuthology.orchestra.run.smithi190.stdout:(25/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.1 MB/s | 839 kB 00:00 2024-05-06T14:19:48.809 INFO:teuthology.orchestra.run.smithi190.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 442 kB/s | 192 kB 00:00 2024-05-06T14:19:48.884 INFO:teuthology.orchestra.run.smithi190.stdout:(27/30): librados2-18.2.0-0.el9.x86_64.rpm 44 MB/s | 3.3 MB 00:00 2024-05-06T14:19:48.911 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-05-06T14:19:48.954 INFO:teuthology.orchestra.run.smithi190.stdout:(28/30): librbd1-18.2.0-0.el9.x86_64.rpm 45 MB/s | 3.0 MB 00:00 2024-05-06T14:19:48.962 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-05-06T14:19:49.054 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-05-06T14:19:49.076 INFO:teuthology.orchestra.run.smithi190.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-05-06T14:19:49.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-05-06T14:19:49.155 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-05-06T14:19:49.217 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-05-06T14:19:49.246 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-05-06T14:19:49.289 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-05-06T14:19:49.355 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-05-06T14:19:49.382 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-05-06T14:19:49.447 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-05-06T14:19:49.562 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-05-06T14:19:49.597 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-05-06T14:19:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-05-06T14:19:49.662 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-05-06T14:19:49.690 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-05-06T14:19:49.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-05-06T14:19:49.758 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-05-06T14:19:50.047 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-05-06T14:19:50.088 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-05-06T14:19:50.178 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-05-06T14:19:50.209 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-05-06T14:19:50.244 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-05-06T14:19:50.360 INFO:teuthology.orchestra.run.smithi190.stdout:(30/30): libarrow-9.0.0-11.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-05-06T14:19:50.361 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:19:50.361 INFO:teuthology.orchestra.run.smithi190.stdout:Total 13 MB/s | 53 MB 00:04 2024-05-06T14:19:50.435 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:19:50.471 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:19:50.471 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:19:50.639 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-05-06T14:19:50.665 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-05-06T14:19:50.761 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-05-06T14:19:50.828 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-05-06T14:19:50.860 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-05-06T14:19:50.891 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:19:50.891 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:19:52.253 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:19:52.335 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-05-06T14:19:52.373 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-05-06T14:19:52.422 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-05-06T14:19:52.732 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-05-06T14:19:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:18.2.0-0.el9.x86_64 5/32 2024-05-06T14:19:52.912 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-05-06T14:19:52.923 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:18.2.0-0.el9.x86_64 5/32 2024-05-06T14:19:52.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-05-06T14:19:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:18.2.0-0.el9.x86_64 6/32 2024-05-06T14:19:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-05-06T14:19:53.103 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:18.2.0-0.el9.x86_64 7/32 2024-05-06T14:19:53.140 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-05-06T14:19:53.202 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 9/32 2024-05-06T14:19:53.266 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-05-06T14:19:53.293 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-05-06T14:19:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:18.2.0-0.el9.x86_64 11/32 2024-05-06T14:19:53.465 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-05-06T14:19:53.483 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-05-06T14:19:53.500 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-05-06T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-05-06T14:19:53.531 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-05-06T14:19:53.556 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-05-06T14:19:53.583 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-05-06T14:19:53.620 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-05-06T14:19:53.634 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-05-06T14:19:53.634 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-06T14:19:53.634 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-06T14:19:53.634 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:19:53.652 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-05-06T14:19:53.661 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-05-06T14:19:53.949 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-05-06T14:19:53.990 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-05-06T14:19:54.080 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:18.2.0-0.el9.x86_64 20/32 2024-05-06T14:19:54.112 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:18.2.0-0.el9.x86_64 21/32 2024-05-06T14:19:54.148 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-05-06T14:19:54.539 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-05-06T14:19:54.560 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-05-06T14:19:54.631 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:18.2.0-0.el9.x86_64 24/32 2024-05-06T14:19:54.698 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:18.2.0-0.el9.x86_64 25/32 2024-05-06T14:19:54.729 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:18.2.0-0.el9.x86_64 26/32 2024-05-06T14:19:56.729 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-05-06T14:19:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-05-06T14:19:57.299 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.2.0-0.el9.x86_64 27/32 2024-05-06T14:19:57.371 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-05-06T14:19:57.465 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:18.2.0-0.el9.x86_64 28/32 2024-05-06T14:19:57.465 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-06T14:19:57.465 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-06T14:19:57.465 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:19:57.488 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-05-06T14:20:03.102 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-05-06T14:20:03.103 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:03.854 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-05-06T14:20:03.878 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-05-06T14:20:03.878 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:03.878 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-06T14:20:03.879 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-06T14:20:03.879 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-06T14:20:03.879 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:03.880 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-06T14:20:03.965 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-06T14:20:03.965 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-05-06T14:20:05.104 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-05-06T14:20:05.105 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-05-06T14:20:05.106 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-05-06T14:20:05.880 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-05-06T14:20:05.880 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:05.880 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-05-06T14:20:05.880 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.880 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:05.880 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: fmt-8.1.1-5.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-06T14:20:05.881 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:05.882 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:06.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-05-06T14:20:06.714 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:24 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:06.837 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:06.838 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:06.838 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:20:06.838 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:06.838 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:06.838 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-05-06T14:20:06.838 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-05-06T14:20:06.839 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:06.840 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 40 M 2024-05-06T14:20:06.840 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 165 M 2024-05-06T14:20:06.840 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:18.2.0-0.el9.x86_64 29/32 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-05-06T14:20:06.868 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-05-06T14:20:06.869 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:07.155 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.5 MB/s | 161 kB 00:00 2024-05-06T14:20:07.324 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-05-06T14:20:07.359 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 312 kB/s | 64 kB 00:00 2024-05-06T14:20:07.580 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-05-06T14:20:07.605 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-0.el9.x86_64 30/32 2024-05-06T14:20:07.605 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:07.605 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-06T14:20:07.605 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-06T14:20:07.605 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-06T14:20:07.605 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:07.607 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-06T14:20:07.691 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-05-06T14:20:07.691 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-06T14:20:07.722 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): ceph-test-18.2.0-0.el9.x86_64.rpm 59 MB/s | 40 MB 00:00 2024-05-06T14:20:07.724 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:07.725 INFO:teuthology.orchestra.run.smithi153.stdout:Total 45 MB/s | 40 MB 00:00 2024-05-06T14:20:07.752 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:07.772 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:07.772 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:08.097 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:08.097 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:08.717 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-06T14:20:08.765 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-06T14:20:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-05-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-05-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:18.2.0-0.el9.x86_64 1/32 2024-05-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:18.2.0-0.el9.x86_64 2/32 2024-05-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:18.2.0-0.el9.x86_64 3/32 2024-05-06T14:20:08.794 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:18.2.0-0.el9.x86_64 4/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:18.2.0-0.el9.x86_64 5/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:18.2.0-0.el9.x86_64 6/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:18.2.0-0.el9.x86_64 7/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:18.2.0-0.el9.x86_64 8/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:18.2.0-0.el9.x86_64 9/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:18.2.0-0.el9.x86_64 10/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:18.2.0-0.el9.x86_64 11/32 2024-05-06T14:20:08.795 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:18.2.0-0.el9.x86_64 12/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:18.2.0-0.el9.x86_64 13/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-05-06T14:20:08.796 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:18.2.0-0.el9.x86_64 29/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-05-06T14:20:08.797 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:18.2.0-0.el9.x86_64 31/32 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:18.2.0-0.el9.x86_64 librbd1-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: fmt-8.1.1-5.el9.x86_64 2024-05-06T14:20:09.506 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:09.508 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:09.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-05-06T14:20:10.311 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:27 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:10.434 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:10.434 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:10.434 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:20:10.434 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:10.434 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:18.2.0-0.el9 ceph 40 M 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:18.2.0-0.el9 ceph 161 k 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-05-06T14:20:10.435 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:10.436 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 40 M 2024-05-06T14:20:10.436 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 165 M 2024-05-06T14:20:10.436 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:10.816 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): libcephsqlite-18.2.0-0.el9.x86_64.rpm 1.5 MB/s | 161 kB 00:00 2024-05-06T14:20:10.934 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-05-06T14:20:10.967 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 422 kB/s | 64 kB 00:00 2024-05-06T14:20:11.394 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): ceph-test-18.2.0-0.el9.x86_64.rpm 58 MB/s | 40 MB 00:00 2024-05-06T14:20:11.396 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:11.396 INFO:teuthology.orchestra.run.smithi190.stdout:Total 42 MB/s | 40 MB 00:00 2024-05-06T14:20:11.423 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:11.442 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:11.443 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:11.767 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:11.767 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:12.291 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:12.377 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-06T14:20:12.434 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-06T14:20:12.457 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-05-06T14:20:12.470 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-05-06T14:20:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-05-06T14:20:14.429 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-05-06T14:20:14.430 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/4 2024-05-06T14:20:14.430 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/4 2024-05-06T14:20:14.430 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:18.2.0-0.el9.x86_64 libcephsqlite-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:14.763 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:15.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-05-06T14:20:15.518 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:33 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:15.644 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:15.646 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:15.646 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-05-06T14:20:15.646 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-06T14:20:15.647 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-06T14:20:15.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-06T14:20:15.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout:Install 33 Packages 2024-05-06T14:20:15.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:15.651 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 M 2024-05-06T14:20:15.651 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 109 M 2024-05-06T14:20:15.651 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:16.145 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:18.2.0-0.el9.x86_64 3/4 2024-05-06T14:20:16.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-05-06T14:20:16.375 INFO:teuthology.orchestra.run.smithi153.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 65 kB/s | 6.4 kB 00:00 2024-05-06T14:20:16.425 INFO:teuthology.orchestra.run.smithi153.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 9.7 MB/s | 1.4 MB 00:00 2024-05-06T14:20:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 9.7 MB/s | 2.1 MB 00:00 2024-05-06T14:20:16.517 INFO:teuthology.orchestra.run.smithi153.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 9.4 MB/s | 240 kB 00:00 2024-05-06T14:20:16.592 INFO:teuthology.orchestra.run.smithi153.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-05-06T14:20:16.809 INFO:teuthology.orchestra.run.smithi153.stdout:(6/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 46 MB/s | 18 MB 00:00 2024-05-06T14:20:16.901 INFO:teuthology.orchestra.run.smithi153.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 660 kB/s | 253 kB 00:00 2024-05-06T14:20:16.960 INFO:teuthology.orchestra.run.smithi153.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 3.3 MB/s | 1.2 MB 00:00 2024-05-06T14:20:17.010 INFO:teuthology.orchestra.run.smithi153.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.5 MB/s | 125 kB 00:00 2024-05-06T14:20:17.035 INFO:teuthology.orchestra.run.smithi153.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 473 kB/s | 106 kB 00:00 2024-05-06T14:20:17.060 INFO:teuthology.orchestra.run.smithi153.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 849 kB/s | 135 kB 00:00 2024-05-06T14:20:17.085 INFO:teuthology.orchestra.run.smithi153.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-05-06T14:20:17.144 INFO:teuthology.orchestra.run.smithi153.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 416 kB/s | 35 kB 00:00 2024-05-06T14:20:17.178 INFO:teuthology.orchestra.run.smithi153.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 554 kB/s | 51 kB 00:00 2024-05-06T14:20:17.203 INFO:teuthology.orchestra.run.smithi153.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-05-06T14:20:17.228 INFO:teuthology.orchestra.run.smithi153.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 496 kB/s | 42 kB 00:00 2024-05-06T14:20:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-05-06T14:20:17.395 INFO:teuthology.orchestra.run.smithi153.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-05-06T14:20:17.429 INFO:teuthology.orchestra.run.smithi153.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 761 kB/s | 172 kB 00:00 2024-05-06T14:20:17.454 INFO:teuthology.orchestra.run.smithi153.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-05-06T14:20:17.480 INFO:teuthology.orchestra.run.smithi153.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-05-06T14:20:17.505 INFO:teuthology.orchestra.run.smithi153.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-05-06T14:20:17.530 INFO:teuthology.orchestra.run.smithi153.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-05-06T14:20:17.555 INFO:teuthology.orchestra.run.smithi153.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-05-06T14:20:17.581 INFO:teuthology.orchestra.run.smithi153.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-05-06T14:20:17.606 INFO:teuthology.orchestra.run.smithi153.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-05-06T14:20:17.631 INFO:teuthology.orchestra.run.smithi153.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-06T14:20:17.657 INFO:teuthology.orchestra.run.smithi153.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-05-06T14:20:17.682 INFO:teuthology.orchestra.run.smithi153.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-05-06T14:20:17.707 INFO:teuthology.orchestra.run.smithi153.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-05-06T14:20:17.757 INFO:teuthology.orchestra.run.smithi153.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-05-06T14:20:17.783 INFO:teuthology.orchestra.run.smithi153.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-05-06T14:20:17.808 INFO:teuthology.orchestra.run.smithi153.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-05-06T14:20:17.810 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:17.810 INFO:teuthology.orchestra.run.smithi153.stdout:Total 14 MB/s | 30 MB 00:02 2024-05-06T14:20:17.929 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:17.966 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:17.966 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:18.100 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:18.2.0-0.el9.x86_64 4/4 2024-05-06T14:20:18.100 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:18.2.0-0.el9.x86_64 1/4 2024-05-06T14:20:18.100 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:18.2.0-0.el9.x86_64 2/4 2024-05-06T14:20:18.100 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-06T14:20:18.249 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:18.251 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:18.2.0-0.el9.x86_64 libcephsqlite-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:18.384 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:18.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-05-06T14:20:19.205 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:36 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:19.329 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:18.2.0-0.el9 ceph 6.4 k 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:18.2.0-0.el9 ceph 2.1 M 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:18.2.0-0.el9 ceph 1.4 M 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:18.2.0-0.el9 ceph-noarch 240 k 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:18.2.0-0.el9 ceph 4.4 M 2024-05-06T14:20:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:18.2.0-0.el9 ceph 18 M 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-06T14:20:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-06T14:20:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-06T14:20:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-06T14:20:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-06T14:20:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-06T14:20:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-06T14:20:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-06T14:20:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout:Install 33 Packages 2024-05-06T14:20:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:19.337 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 M 2024-05-06T14:20:19.337 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 109 M 2024-05-06T14:20:19.337 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:19.402 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:19.477 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-05-06T14:20:19.509 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-05-06T14:20:19.542 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-05-06T14:20:19.635 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-05-06T14:20:19.712 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-05-06T14:20:19.762 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-05-06T14:20:19.796 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-05-06T14:20:19.826 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-05-06T14:20:19.837 INFO:teuthology.orchestra.run.smithi190.stdout:(1/33): ceph-18.2.0-0.el9.x86_64.rpm 57 kB/s | 6.4 kB 00:00 2024-05-06T14:20:19.861 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-05-06T14:20:19.887 INFO:teuthology.orchestra.run.smithi190.stdout:(2/33): ceph-mgr-18.2.0-0.el9.x86_64.rpm 8.8 MB/s | 1.4 MB 00:00 2024-05-06T14:20:19.894 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-05-06T14:20:19.946 INFO:teuthology.orchestra.run.smithi190.stdout:(3/33): ceph-mds-18.2.0-0.el9.x86_64.rpm 9.5 MB/s | 2.1 MB 00:00 2024-05-06T14:20:19.960 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-05-06T14:20:19.971 INFO:teuthology.orchestra.run.smithi190.stdout:(4/33): ceph-mgr-modules-core-18.2.0-0.el9.noar 9.4 MB/s | 240 kB 00:00 2024-05-06T14:20:19.995 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-05-06T14:20:20.029 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-05-06T14:20:20.063 INFO:teuthology.orchestra.run.smithi190.stdout:(5/33): ceph-mon-18.2.0-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-05-06T14:20:20.067 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-05-06T14:20:20.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-05-06T14:20:20.288 INFO:teuthology.orchestra.run.smithi190.stdout:(6/33): ceph-osd-18.2.0-0.el9.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-05-06T14:20:20.305 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-05-06T14:20:20.385 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-05-06T14:20:20.437 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-05-06T14:20:20.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-05-06T14:20:20.533 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-05-06T14:20:20.589 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-05-06T14:20:20.647 INFO:teuthology.orchestra.run.smithi190.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 375 kB/s | 253 kB 00:00 2024-05-06T14:20:20.714 INFO:teuthology.orchestra.run.smithi190.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 250 kB/s | 106 kB 00:00 2024-05-06T14:20:20.756 INFO:teuthology.orchestra.run.smithi190.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-05-06T14:20:20.777 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-05-06T14:20:20.821 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-05-06T14:20:20.864 INFO:teuthology.orchestra.run.smithi190.stdout:(10/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-05-06T14:20:20.898 INFO:teuthology.orchestra.run.smithi190.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 679 kB/s | 125 kB 00:00 2024-05-06T14:20:20.900 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-05-06T14:20:20.998 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-05-06T14:20:21.037 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-05-06T14:20:21.098 INFO:teuthology.orchestra.run.smithi190.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-05-06T14:20:21.152 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-05-06T14:20:21.305 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-05-06T14:20:21.330 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-05-06T14:20:21.355 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-05-06T14:20:21.355 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:21.355 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-06T14:20:21.356 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-06T14:20:21.356 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-06T14:20:21.356 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:21.515 INFO:teuthology.orchestra.run.smithi190.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 56 kB/s | 35 kB 00:00 2024-05-06T14:20:21.657 INFO:teuthology.orchestra.run.smithi190.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 217 kB/s | 172 kB 00:00 2024-05-06T14:20:21.797 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] python3-bcrypt-3.2.2-1.el9.x86_64.rpm: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-bcrypt-3.2.2-1.el9.x86_64.rpm [SSL certificate problem: certificate has expired] 2024-05-06T14:20:21.841 INFO:teuthology.orchestra.run.smithi190.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 74 kB/s | 51 kB 00:00 2024-05-06T14:20:21.962 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] python3-cheroot-8.6.0-4.el9.noarch.rpm: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-cheroot-8.6.0-4.el9.noarch.rpm [SSL certificate problem: certificate has expired] 2024-05-06T14:20:22.158 INFO:teuthology.orchestra.run.smithi190.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 65 kB/s | 42 kB 00:00 2024-05-06T14:20:22.274 INFO:teuthology.orchestra.run.smithi190.stdout:[MIRROR] python3-cherrypy-18.6.1-2.el9.noarch.rpm: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-cherrypy-18.6.1-2.el9.noarch.rpm [SSL certificate problem: certificate has expired] 2024-05-06T14:20:22.614 INFO:teuthology.orchestra.run.smithi190.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 46 kB/s | 43 kB 00:00 2024-05-06T14:20:22.932 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-05-06T14:20:22.960 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-05-06T14:20:22.961 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:22.961 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-06T14:20:22.961 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-06T14:20:22.961 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-06T14:20:22.961 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:23.074 INFO:teuthology.orchestra.run.smithi190.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 139 kB/s | 172 kB 00:01 2024-05-06T14:20:23.408 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-05-06T14:20:23.431 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-05-06T14:20:23.431 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:23.432 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-06T14:20:23.432 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-06T14:20:23.432 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-06T14:20:23.432 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:23.566 INFO:teuthology.orchestra.run.smithi190.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 254 kB/s | 358 kB 00:01 2024-05-06T14:20:23.633 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-05-06T14:20:23.656 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-05-06T14:20:23.657 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:23.657 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-06T14:20:23.657 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-06T14:20:23.657 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-06T14:20:23.657 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:23.666 INFO:teuthology.orchestra.run.smithi190.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 30 kB/s | 18 kB 00:00 2024-05-06T14:20:23.720 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-05-06T14:20:24.042 INFO:teuthology.orchestra.run.smithi190.stdout:(21/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 7.4 kB/s | 11 kB 00:01 2024-05-06T14:20:24.150 INFO:teuthology.orchestra.run.smithi190.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 40 kB/s | 23 kB 00:00 2024-05-06T14:20:24.226 INFO:teuthology.orchestra.run.smithi190.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 35 kB/s | 19 kB 00:00 2024-05-06T14:20:24.626 INFO:teuthology.orchestra.run.smithi190.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 34 kB/s | 20 kB 00:00 2024-05-06T14:20:24.884 INFO:teuthology.orchestra.run.smithi190.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 63 kB/s | 46 kB 00:00 2024-05-06T14:20:25.076 INFO:teuthology.orchestra.run.smithi190.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 93 kB/s | 79 kB 00:00 2024-05-06T14:20:25.325 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-05-06T14:20:25.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-05-06T14:20:25.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-05-06T14:20:25.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-05-06T14:20:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-05-06T14:20:25.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-05-06T14:20:25.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-05-06T14:20:25.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-05-06T14:20:25.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-05-06T14:20:25.460 INFO:teuthology.orchestra.run.smithi190.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 29 kB/s | 16 kB 00:00 2024-05-06T14:20:25.894 INFO:teuthology.orchestra.run.smithi190.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 215 kB/s | 272 kB 00:01 2024-05-06T14:20:25.985 INFO:teuthology.orchestra.run.smithi190.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 99 kB/s | 90 kB 00:00 2024-05-06T14:20:26.119 INFO:teuthology.orchestra.run.smithi190.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 54 kB/s | 36 kB 00:00 2024-05-06T14:20:26.166 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-05-06T14:20:26.166 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:26.166 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:26.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:26.166 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-06T14:20:26.167 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-06T14:20:26.168 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-06T14:20:26.169 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-06T14:20:26.169 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:26.169 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:26.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-05-06T14:20:26.702 INFO:teuthology.orchestra.run.smithi190.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 34 kB/s | 20 kB 00:00 2024-05-06T14:20:27.011 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:45 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 195 kB/s | 230 kB 00:01 2024-05-06T14:20:27.090 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:20:27.131 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:27.132 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:20:27.132 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:27.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-05-06T14:20:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 328 kB/s | 427 kB 00:01 2024-05-06T14:20:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.8 MB/s | 30 MB 00:07 2024-05-06T14:20:27.405 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:27.443 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:27.443 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:45 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:27.824 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:27.825 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:20:27.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:27.826 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 209 k 2024-05-06T14:20:27.826 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 200 k 2024-05-06T14:20:27.826 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:27.882 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-18.2.0-0.el9.noarch.rpm 3.7 MB/s | 209 kB 00:00 2024-05-06T14:20:27.882 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:27.883 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.6 MB/s | 209 kB 00:00 2024-05-06T14:20:27.883 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:27.888 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:27.888 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:27.890 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:27.890 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:27.955 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:28.374 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:28.458 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:28.771 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:28.881 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-05-06T14:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-05-06T14:20:29.002 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:29.003 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:29.003 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:29.003 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-05-06T14:20:29.003 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:29.003 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-05-06T14:20:29.105 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-05-06T14:20:29.183 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-05-06T14:20:29.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-05-06T14:20:29.223 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-05-06T14:20:29.266 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-05-06T14:20:29.296 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-05-06T14:20:29.322 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-05-06T14:20:29.356 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-05-06T14:20:29.397 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-05-06T14:20:29.431 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-05-06T14:20:29.466 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-05-06T14:20:29.504 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-05-06T14:20:29.578 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-05-06T14:20:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-05-06T14:20:29.733 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:47 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-05-06T14:20:29.853 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 142 k 2024-05-06T14:20:29.854 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 399 k 2024-05-06T14:20:29.855 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:29.866 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-05-06T14:20:29.904 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-05-06T14:20:29.914 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 2.4 MB/s | 142 kB 00:00 2024-05-06T14:20:29.915 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:29.915 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.3 MB/s | 142 kB 00:00 2024-05-06T14:20:29.915 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:29.923 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:29.923 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:29.961 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-05-06T14:20:29.981 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:29.982 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-05-06T14:20:30.126 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:30.191 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:20:30.204 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-05-06T14:20:30.214 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:20:30.214 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:30.215 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-06T14:20:30.215 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:30.259 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-05-06T14:20:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-05-06T14:20:30.434 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-05-06T14:20:30.474 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-05-06T14:20:30.590 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-05-06T14:20:30.743 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 28/33 2024-05-06T14:20:30.767 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-05-06T14:20:30.792 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:18.2.0-0.el9.x86_64 29/33 2024-05-06T14:20:30.792 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:30.792 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-06T14:20:30.792 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-06T14:20:30.792 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-06T14:20:30.792 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:31.085 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:20:31.085 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:31.085 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:31.085 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:31.085 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:31.085 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:31.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-05-06T14:20:31.791 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:49 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:31.870 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:20:31.913 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:31.914 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:20:31.914 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:31.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-05-06T14:20:32.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-05-06T14:20:32.388 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:18.2.0-0.el9.x86_64 30/33 2024-05-06T14:20:32.388 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:32.388 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-06T14:20:32.389 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-06T14:20:32.389 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-06T14:20:32.389 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:32.489 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:50 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:32.609 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-05-06T14:20:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-05-06T14:20:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:32.612 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-05-06T14:20:32.612 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 8.1 M 2024-05-06T14:20:32.612 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:32.768 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 209 kB/s | 23 kB 00:00 2024-05-06T14:20:32.793 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 107 kB/s | 15 kB 00:00 2024-05-06T14:20:32.837 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-05-06T14:20:32.844 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 9.1 MB/s | 1.7 MB 00:00 2024-05-06T14:20:32.859 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:18.2.0-0.el9.x86_64 31/33 2024-05-06T14:20:32.859 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:32.859 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-06T14:20:32.859 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-06T14:20:32.859 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-06T14:20:32.860 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:32.911 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 64 kB/s | 9.0 kB 00:00 2024-05-06T14:20:33.011 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 184 kB/s | 31 kB 00:00 2024-05-06T14:20:33.036 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 169 kB/s | 41 kB 00:00 2024-05-06T14:20:33.062 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-05-06T14:20:33.084 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:18.2.0-0.el9.x86_64 32/33 2024-05-06T14:20:33.084 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:33.084 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-06T14:20:33.085 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-06T14:20:33.085 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-06T14:20:33.085 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:33.148 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:18.2.0-0.el9.x86_64 33/33 2024-05-06T14:20:33.161 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 753 kB/s | 188 kB 00:00 2024-05-06T14:20:33.163 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:33.164 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2024-05-06T14:20:33.164 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:33.172 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:33.221 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:33.222 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:33.509 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:33.571 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-05-06T14:20:33.605 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-05-06T14:20:33.651 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-05-06T14:20:33.692 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-05-06T14:20:33.720 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-05-06T14:20:33.948 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-05-06T14:20:33.982 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-05-06T14:20:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-05-06T14:20:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-05-06T14:20:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-05-06T14:20:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-05-06T14:20:34.482 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-05-06T14:20:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-05-06T14:20:34.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-05-06T14:20:34.736 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-05-06T14:20:34.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-06T14:20:34.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-06T14:20:34.737 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:34.737 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:34.763 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:18.2.0-0.el9.x86_64 33/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:18.2.0-0.el9.x86_64 1/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:18.2.0-0.el9.x86_64 2/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:18.2.0-0.el9.x86_64 3/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:18.2.0-0.el9.x86_64 4/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:18.2.0-0.el9.x86_64 5/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 6/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-05-06T14:20:34.764 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-05-06T14:20:34.765 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-05-06T14:20:34.766 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-05-06T14:20:34.766 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-05-06T14:20:34.766 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-05-06T14:20:34.766 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-05-06T14:20:34.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-06T14:20:35.446 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:35.567 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:35.568 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:35.568 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-06T14:20:35.569 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-05-06T14:20:35.570 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:18.2.0-0.el9.noarch 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-06T14:20:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-06T14:20:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:35.573 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 38 M 2024-05-06T14:20:35.574 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-05-06T14:20:35.574 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:35.793 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.0-0. 50 MB/s | 7.4 MB 00:00 2024-05-06T14:20:35.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-05-06T14:20:35.910 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 742 kB/s | 195 kB 00:00 2024-05-06T14:20:35.977 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-05-06T14:20:36.386 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-05-06T14:20:36.422 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:53 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:36.499 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:20:36.511 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 34 kB/s | 18 kB 00:00 2024-05-06T14:20:36.540 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:36.541 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:20:36.541 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:36.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-05-06T14:20:36.970 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 60 kB/s | 35 kB 00:00 2024-05-06T14:20:37.045 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-05-06T14:20:37.109 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:54 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:37.228 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:18.2.0-0.el9 ceph-noarch 209 k 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:20:37.229 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:37.230 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 209 k 2024-05-06T14:20:37.230 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 200 k 2024-05-06T14:20:37.230 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:37.294 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-18.2.0-0.el9.noarch.rpm 3.2 MB/s | 209 kB 00:00 2024-05-06T14:20:37.295 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:37.295 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.1 MB/s | 209 kB 00:00 2024-05-06T14:20:37.295 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:37.300 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:37.300 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:37.302 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:37.303 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:37.370 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:37.695 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.0 MB/s | 4.7 MB 00:01 2024-05-06T14:20:37.803 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 293 kB/s | 244 kB 00:00 2024-05-06T14:20:37.828 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:37.903 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:38.190 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:38.254 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-05-06T14:20:38.434 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:18.2.0-0.el9.noarch 1/1 2024-05-06T14:20:38.434 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:38.434 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:38.434 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:18.2.0-0.el9.noarch 2024-05-06T14:20:38.434 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:38.434 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:38.562 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 289 kB/s | 250 kB 00:00 2024-05-06T14:20:38.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-05-06T14:20:39.136 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:56 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:39.257 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-0.el9 ceph 142 k 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:39.258 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:39.259 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:20:39.259 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:39.259 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 142 k 2024-05-06T14:20:39.259 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 399 k 2024-05-06T14:20:39.259 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:39.331 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-18.2.0-0.el9.x86_64 1.9 MB/s | 142 kB 00:00 2024-05-06T14:20:39.332 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:39.332 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.9 MB/s | 142 kB 00:00 2024-05-06T14:20:39.332 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:39.340 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:39.340 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:39.397 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:39.397 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:39.482 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.7 MB/s | 16 MB 00:01 2024-05-06T14:20:39.482 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:39.482 INFO:teuthology.orchestra.run.smithi153.stdout:Total 9.7 MB/s | 38 MB 00:03 2024-05-06T14:20:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:39.634 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:20:39.658 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:20:39.658 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:20:39.659 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-06T14:20:39.659 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:39.837 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:39.863 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:39.863 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:40.268 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:40.269 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:40.505 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:20:40.505 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:40.506 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:40.506 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:18.2.0-0.el9.x86_64 2024-05-06T14:20:40.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:40.506 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:40.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-05-06T14:20:41.017 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:41.075 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-06T14:20:41.119 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-06T14:20:41.223 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-06T14:20:41.224 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:58 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:41.270 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-06T14:20:41.305 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:20:41.348 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:41.349 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:20:41.349 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:41.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-05-06T14:20:41.449 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-06T14:20:41.477 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-06T14:20:41.554 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-06T14:20:41.930 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:58 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:42.047 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:42.047 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:18.2.0-0.el9 ceph-noarch 1.7 M 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:18.2.0-0.el9 ceph-noarch 23 k 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:18.2.0-0.el9 ceph-noarch 15 k 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-05-06T14:20:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-05-06T14:20:42.049 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:42.049 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:42.049 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:42.049 INFO:teuthology.orchestra.run.smithi190.stdout:Install 7 Packages 2024-05-06T14:20:42.049 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:42.050 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.0 M 2024-05-06T14:20:42.050 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 8.1 M 2024-05-06T14:20:42.050 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:42.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-06T14:20:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-06T14:20:42.511 INFO:teuthology.orchestra.run.smithi190.stdout:(1/7): ceph-grafana-dashboards-18.2.0-0.el9.noa 199 kB/s | 23 kB 00:00 2024-05-06T14:20:42.536 INFO:teuthology.orchestra.run.smithi190.stdout:(2/7): ceph-prometheus-alerts-18.2.0-0.el9.noar 103 kB/s | 15 kB 00:00 2024-05-06T14:20:42.587 INFO:teuthology.orchestra.run.smithi190.stdout:(3/7): ceph-mgr-dashboard-18.2.0-0.el9.noarch.r 8.9 MB/s | 1.7 MB 00:00 2024-05-06T14:20:42.703 INFO:teuthology.orchestra.run.smithi190.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2024-05-06T14:20:42.779 INFO:teuthology.orchestra.run.smithi190.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 169 kB/s | 41 kB 00:00 2024-05-06T14:20:42.804 INFO:teuthology.orchestra.run.smithi190.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 181 kB/s | 31 kB 00:00 2024-05-06T14:20:42.871 INFO:teuthology.orchestra.run.smithi190.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-05-06T14:20:42.875 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:42.875 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-05-06T14:20:42.875 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:42.883 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:42.883 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:42.933 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:42.933 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:43.224 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:43.281 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-05-06T14:20:43.315 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-05-06T14:20:43.386 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-05-06T14:20:43.426 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-05-06T14:20:43.455 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 5/7 2024-05-06T14:20:43.550 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-06T14:20:43.686 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 6/7 2024-05-06T14:20:43.716 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-05-06T14:20:44.159 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 7/7 2024-05-06T14:20:44.159 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 1/7 2024-05-06T14:20:44.159 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2/7 2024-05-06T14:20:44.159 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 3/7 2024-05-06T14:20:44.159 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-05-06T14:20:44.159 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-05-06T14:20:44.160 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-05-06T14:20:44.409 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-05-06T14:20:44.409 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:44.409 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:44.409 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:18.2.0-0.el9.noarch 2024-05-06T14:20:44.409 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:18.2.0-0.el9.noarch 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:18.2.0-0.el9.noarch 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:44.585 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-06T14:20:44.619 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-05-06T14:20:44.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-06T14:20:45.134 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:02 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:45.254 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:45.255 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:45.255 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-0.el9 ceph-noarch 7.4 M 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-06T14:20:45.256 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-05-06T14:20:45.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:45.258 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 38 M 2024-05-06T14:20:45.259 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 209 M 2024-05-06T14:20:45.259 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:45.695 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.0-0. 48 MB/s | 7.4 MB 00:00 2024-05-06T14:20:45.753 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 921 kB/s | 195 kB 00:00 2024-05-06T14:20:45.803 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-05-06T14:20:45.937 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 135 kB/s | 33 kB 00:00 2024-05-06T14:20:45.970 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 109 kB/s | 18 kB 00:00 2024-05-06T14:20:46.079 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 248 kB/s | 35 kB 00:00 2024-05-06T14:20:46.340 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 937 kB/s | 244 kB 00:00 2024-05-06T14:20:46.390 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-05-06T14:20:46.457 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-05-06T14:20:46.524 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 8.6 MB/s | 4.7 MB 00:00 2024-05-06T14:20:46.824 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-05-06T14:20:47.125 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-05-06T14:20:47.125 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:47.125 INFO:teuthology.orchestra.run.smithi190.stdout:Total 20 MB/s | 38 MB 00:01 2024-05-06T14:20:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-05-06T14:20:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-06T14:20:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-06T14:20:47.472 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:47.498 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:47.499 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:47.660 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-06T14:20:47.661 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-06T14:20:47.662 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-06T14:20:47.662 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:47.662 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:47.902 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:47.903 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:48.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-05-06T14:20:48.530 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:06 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:48.647 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:48.647 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:48.647 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-06T14:20:48.648 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.2 M 2024-05-06T14:20:48.650 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-05-06T14:20:48.650 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:48.655 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:48.712 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-06T14:20:48.755 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-06T14:20:48.860 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-06T14:20:48.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-06T14:20:48.911 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 803 kB/s | 47 kB 00:00 2024-05-06T14:20:49.075 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-06T14:20:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-06T14:20:49.136 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-05-06T14:20:49.162 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 83 kB/s | 26 kB 00:00 2024-05-06T14:20:49.181 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-06T14:20:49.528 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 405 kB/s | 159 kB 00:00 2024-05-06T14:20:49.554 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 346 kB/s | 222 kB 00:00 2024-05-06T14:20:49.671 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 377 kB/s | 54 kB 00:00 2024-05-06T14:20:49.696 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 522 kB/s | 279 kB 00:00 2024-05-06T14:20:49.763 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-05-06T14:20:49.788 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 129 kB/s | 15 kB 00:00 2024-05-06T14:20:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-06T14:20:49.896 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.0 MB/s | 206 kB 00:00 2024-05-06T14:20:49.979 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-06T14:20:50.072 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-05-06T14:20:50.097 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 191 kB/s | 59 kB 00:00 2024-05-06T14:20:50.122 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 398 kB/s | 90 kB 00:00 2024-05-06T14:20:50.123 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:50.124 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-05-06T14:20:50.194 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:50.209 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:50.209 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:50.294 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:50.295 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:51.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-06T14:20:51.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-06T14:20:51.130 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-06T14:20:51.167 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-06T14:20:51.208 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-06T14:20:51.248 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-06T14:20:51.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-06T14:20:51.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-06T14:20:51.467 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-06T14:20:52.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-06T14:20:52.232 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-06T14:20:52.265 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-06T14:20:52.285 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-06T14:20:52.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-06T14:20:52.312 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-05-06T14:20:52.335 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-05-06T14:20:53.067 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-05-06T14:20:53.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-05-06T14:20:53.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-06T14:20:53.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-06T14:20:53.067 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-06T14:20:53.068 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-06T14:20:53.385 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:53.386 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:53.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-05-06T14:20:54.172 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:12 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:54.292 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:54.293 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:54.293 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-05-06T14:20:54.293 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:54.293 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:54.293 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-05-06T14:20:54.294 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:54.295 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 7.0 M 2024-05-06T14:20:54.295 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-05-06T14:20:54.295 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:54.583 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.9 MB/s | 127 kB 00:00 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 12/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarc 1/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-06T14:20:54.706 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-06T14:20:54.707 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-06T14:20:54.707 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-06T14:20:54.967 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-05-06T14:20:55.017 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-05-06T14:20:55.067 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:18.2.0-0.el9.noarch 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-06T14:20:55.071 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-06T14:20:55.072 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-06T14:20:55.072 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-06T14:20:55.072 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-06T14:20:55.072 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-06T14:20:55.072 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:55.072 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:20:55.159 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 387 kB/s | 248 kB 00:00 2024-05-06T14:20:55.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-05-06T14:20:55.835 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-05-06T14:20:55.836 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:55.836 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-05-06T14:20:55.907 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:55.919 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:55.919 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:55.968 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:12 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:20:56.014 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:56.015 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:56.088 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:18.2.0-0.el9 ceph-noarch 47 k 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:20:56.090 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-06T14:20:56.091 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:56.092 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:20:56.092 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:20:56.092 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-05-06T14:20:56.092 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:20:56.092 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.2 M 2024-05-06T14:20:56.093 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-05-06T14:20:56.093 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:20:56.314 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:56.452 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): ceph-mgr-rook-18.2.0-0.el9.noarch.rpm 719 kB/s | 47 kB 00:00 2024-05-06T14:20:56.474 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-06T14:20:56.494 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 177 kB/s | 19 kB 00:00 2024-05-06T14:20:56.519 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-06T14:20:56.520 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 26 kB 00:00 2024-05-06T14:20:56.570 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-05-06T14:20:56.612 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-05-06T14:20:56.637 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 803 kB/s | 54 kB 00:00 2024-05-06T14:20:56.662 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.9 MB/s | 279 kB 00:00 2024-05-06T14:20:56.704 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 223 kB/s | 15 kB 00:00 2024-05-06T14:20:56.730 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-05-06T14:20:56.797 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 887 kB/s | 59 kB 00:00 2024-05-06T14:20:56.847 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 1.1 MB/s | 206 kB 00:00 2024-05-06T14:20:56.872 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-05-06T14:20:57.043 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-06T14:20:57.106 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 290 kB/s | 90 kB 00:00 2024-05-06T14:20:57.106 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:57.107 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-05-06T14:20:57.130 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-06T14:20:57.168 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:20:57.182 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:20:57.182 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:20:57.190 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-06T14:20:57.223 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-05-06T14:20:57.264 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:20:57.264 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:20:57.846 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:20:58.000 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-05-06T14:20:58.000 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-05-06T14:20:58.000 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-06T14:20:58.000 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-06T14:20:58.000 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-06T14:20:58.000 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-06T14:20:58.010 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-06T14:20:58.065 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-06T14:20:58.129 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-06T14:20:58.166 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-06T14:20:58.207 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-06T14:20:58.247 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-06T14:20:58.247 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:58.247 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:20:58.247 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-05-06T14:20:58.247 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-06T14:20:58.248 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-06T14:20:58.248 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-06T14:20:58.248 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-06T14:20:58.248 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-06T14:20:58.248 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:58.248 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:20:58.302 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-06T14:20:58.402 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-06T14:20:58.450 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-06T14:20:58.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-05-06T14:20:58.974 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:16 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:20:59.093 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:20:59.093 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-05-06T14:20:59.094 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:20:59.095 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 915 k 2024-05-06T14:20:59.095 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.7 M 2024-05-06T14:20:59.095 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:20:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-06T14:20:59.206 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-06T14:20:59.239 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-06T14:20:59.284 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-06T14:20:59.309 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-05-06T14:20:59.436 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 10 MB/s | 835 kB 00:00 2024-05-06T14:20:59.486 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 619 kB/s | 80 kB 00:00 2024-05-06T14:20:59.487 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:20:59.487 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.3 MB/s | 915 kB 00:00 2024-05-06T14:20:59.498 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:20:59.520 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:20:59.521 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:20:59.600 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:20:59.601 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:20:59.784 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:20:59.917 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-06T14:20:59.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-05-06T14:21:00.013 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-0.el9.noarch 13/13 2024-05-06T14:21:00.013 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:18.2.0-0.el9.noarch 1/13 2024-05-06T14:21:00.013 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-06T14:21:00.014 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-06T14:21:00.319 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-06T14:21:00.319 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:00.319 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:00.319 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:18.2.0-0.el9.noarch 2024-05-06T14:21:00.319 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-06T14:21:00.320 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-06T14:21:00.321 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-06T14:21:00.321 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:00.321 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:00.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-05-06T14:21:00.649 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-05-06T14:21:00.650 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-05-06T14:21:00.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-06T14:21:00.914 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:00.915 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:21:00.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-06T14:21:00.915 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:00.915 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:01.094 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:18 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:01.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-05-06T14:21:01.213 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:18.2.0-0.el9 ceph-noarch 127 k 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-06T14:21:01.214 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 7.0 M 2024-05-06T14:21:01.216 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-05-06T14:21:01.216 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:01.708 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:19 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:01.790 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): ceph-mgr-cephadm-18.2.0-0.el9.noarch.rpm 1.9 MB/s | 127 kB 00:00 2024-05-06T14:21:01.832 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:21:01.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:01.834 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 126 k 2024-05-06T14:21:01.834 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-05-06T14:21:01.834 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:21:01.891 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 2.2 MB/s | 126 kB 00:00 2024-05-06T14:21:01.891 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:01.892 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.1 MB/s | 126 kB 00:00 2024-05-06T14:21:01.892 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:21:01.897 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:21:01.897 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:21:01.962 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:21:01.962 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:21:02.048 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 762 kB/s | 248 kB 00:00 2024-05-06T14:21:02.073 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-05-06T14:21:02.107 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 994 kB/s | 58 kB 00:00 2024-05-06T14:21:02.109 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:21:02.132 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-05-06T14:21:02.203 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:02.574 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.0 MB/s | 6.0 MB 00:00 2024-05-06T14:21:02.575 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:02.576 INFO:teuthology.orchestra.run.smithi190.stdout:Total 5.1 MB/s | 7.0 MB 00:01 2024-05-06T14:21:02.601 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:02.645 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:02.656 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:02.656 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:02.750 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:02.750 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:02.856 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:02.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:21:02.856 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:02.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:02.857 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:03.044 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:03.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-05-06T14:21:03.212 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-06T14:21:03.275 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-06T14:21:03.600 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:21 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:03.680 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:03.720 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:03.721 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:03.721 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:03.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-05-06T14:21:03.792 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-06T14:21:03.878 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-06T14:21:03.937 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-06T14:21:03.962 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-05-06T14:21:04.320 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:22 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:04.441 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:21:04.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:04.443 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 k 2024-05-06T14:21:04.443 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 137 k 2024-05-06T14:21:04.443 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:21:04.511 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 449 kB/s | 30 kB 00:00 2024-05-06T14:21:04.511 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:04.511 INFO:teuthology.orchestra.run.smithi153.stdout:Total 441 kB/s | 30 kB 00:00 2024-05-06T14:21:04.512 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:21:04.515 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:21:04.515 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:21:04.517 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:21:04.517 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:21:04.606 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:21:04.705 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:04.737 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 6/6 2024-05-06T14:21:04.738 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 1/6 2024-05-06T14:21:04.738 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-06T14:21:04.738 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-06T14:21:04.738 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-06T14:21:04.738 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-06T14:21:04.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-06T14:21:04.995 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:04.995 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:04.995 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:18.2.0-0.el9.noarch 2024-05-06T14:21:04.995 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-06T14:21:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-06T14:21:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-06T14:21:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-06T14:21:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-06T14:21:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:04.996 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:05.034 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:05.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-05-06T14:21:05.245 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:05.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:05.246 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:21:05.246 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:05.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:05.246 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:05.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-05-06T14:21:05.769 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:22 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:05.888 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:18.2.0-0.el9 ceph 835 k 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:05.889 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:05.890 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-05-06T14:21:05.890 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:05.891 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 915 k 2024-05-06T14:21:05.891 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.7 M 2024-05-06T14:21:05.891 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:05.919 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:23 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:06.000 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:06.001 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-fuse-18.2.0-0.el9.x86_64.rpm 11 MB/s | 835 kB 00:00 2024-05-06T14:21:06.041 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:06.043 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:06.043 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:06.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-05-06T14:21:06.376 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 178 kB/s | 80 kB 00:00 2024-05-06T14:21:06.377 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:06.377 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.8 MB/s | 915 kB 00:00 2024-05-06T14:21:06.389 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:06.410 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:06.410 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:06.491 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:06.491 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:06.622 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:24 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:06.675 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:06.704 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:06.745 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:06.746 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:06.746 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:06.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-05-06T14:21:06.806 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-06T14:21:06.873 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-05-06T14:21:07.319 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:25 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:07.399 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:07.439 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:07.440 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:07.440 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:07.480 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:18.2.0-0.el9.x86_64 2/2 2024-05-06T14:21:07.480 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:18.2.0-0.el9.x86_64 1/2 2024-05-06T14:21:07.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-05-06T14:21:07.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-06T14:21:07.734 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:07.734 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:07.735 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:18.2.0-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-06T14:21:07.735 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:07.735 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:07.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-05-06T14:21:08.017 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:26 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:08.095 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:08.135 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:08.136 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:08.136 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:08.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-05-06T14:21:08.514 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:25 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:08.637 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:08.637 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:08.637 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:18.2.0-0.el9 ceph 126 k 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 126 k 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-05-06T14:21:08.638 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:08.704 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-18.2.0-0.el9.x86_64.rpm 1.9 MB/s | 126 kB 00:00 2024-05-06T14:21:08.705 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:08.706 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.8 MB/s | 126 kB 00:00 2024-05-06T14:21:08.706 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:08.711 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:26 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:08.712 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:08.712 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:08.773 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:08.773 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:08.790 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:08.830 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:08.831 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:08.832 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:08.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-05-06T14:21:08.932 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:09.063 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:09.404 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:27 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:09.439 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:09.483 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:09.523 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:09.524 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-05-06T14:21:09.524 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:09.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-05-06T14:21:09.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:09.644 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:09.644 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:09.644 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:09.644 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:09.644 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:09.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-05-06T14:21:10.101 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:28 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:10.224 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:10.225 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:10.225 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 86 k 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 230 k 2024-05-06T14:21:10.226 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:21:10.297 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-05-06T14:21:10.298 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:10.298 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-05-06T14:21:10.298 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:21:10.305 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:21:10.305 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:21:10.368 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:21:10.368 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:21:10.425 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:27 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:10.506 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:10.509 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:21:10.549 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:10.550 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:10.550 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:10.609 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:10.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-05-06T14:21:10.921 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:11.132 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:28 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:11.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:11.162 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:11.163 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:21:11.163 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:11.163 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:11.163 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:11.251 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:11.251 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:11.251 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:18.2.0-0.el9 ceph 30 k 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:21:11.252 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:11.253 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 k 2024-05-06T14:21:11.253 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 137 k 2024-05-06T14:21:11.253 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:11.316 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-18.2.0-0.el9.x86_64.rpm 484 kB/s | 30 kB 00:00 2024-05-06T14:21:11.316 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:11.316 INFO:teuthology.orchestra.run.smithi190.stdout:Total 474 kB/s | 30 kB 00:00 2024-05-06T14:21:11.317 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:11.320 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:11.320 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:11.322 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:11.323 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:11.413 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:11.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-05-06T14:21:11.519 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:11.838 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:11.947 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:12.032 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:12.032 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:12.033 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:12.033 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:12.033 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:12.033 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:12.067 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.0 M 2024-05-06T14:21:12.068 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 12 M 2024-05-06T14:21:12.069 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:21:12.168 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 30 MB/s | 3.0 MB 00:00 2024-05-06T14:21:12.168 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:12.169 INFO:teuthology.orchestra.run.smithi153.stdout:Total 30 MB/s | 3.0 MB 00:00 2024-05-06T14:21:12.169 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:21:12.178 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:21:12.178 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:21:12.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-05-06T14:21:12.262 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:21:12.262 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:21:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:21:12.706 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:29 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:12.785 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:12.811 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:12.825 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:12.826 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:12.827 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:12.835 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:12.835 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:21:12.836 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-06T14:21:12.836 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-06T14:21:12.836 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-06T14:21:12.836 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:12.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-05-06T14:21:13.405 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:30 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:13.487 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:13.524 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:13.525 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:13.525 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:13.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-05-06T14:21:13.773 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:13.773 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:13.773 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:21:13.773 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:13.773 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:13.773 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:13.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-05-06T14:21:14.103 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:31 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:14.181 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:14.221 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:14.222 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:14.222 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:14.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-05-06T14:21:14.501 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:32 ago on Mon 06 May 2024 02:19:42 PM UTC. 2024-05-06T14:21:14.628 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-05-06T14:21:14.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:14.630 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 169 k 2024-05-06T14:21:14.630 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 490 k 2024-05-06T14:21:14.630 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-05-06T14:21:14.689 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2024-05-06T14:21:14.689 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:14.689 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.7 MB/s | 169 kB 00:00 2024-05-06T14:21:14.690 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-05-06T14:21:14.697 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-05-06T14:21:14.697 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-05-06T14:21:14.760 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-05-06T14:21:14.760 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-05-06T14:21:14.808 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:31 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:14.886 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:14.910 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-05-06T14:21:14.926 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:14.928 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:14.928 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:14.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-05-06T14:21:15.003 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:15.514 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:32 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:15.594 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:15.628 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:15.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:15.628 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-05-06T14:21:15.628 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:15.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:21:15.628 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-05-06T14:21:15.634 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:15.636 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:15.636 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:15.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-05-06T14:21:15.869 DEBUG:teuthology.parallel:result is None 2024-05-06T14:21:16.224 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:33 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:16.302 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:18.2.0-0.el9.x86_64 is already installed. 2024-05-06T14:21:16.343 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:16.344 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-05-06T14:21:16.344 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:16.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-05-06T14:21:16.920 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:33 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:17.039 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:18.2.0-0.el9 ceph 86 k 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:17.040 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:21:17.041 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:17.041 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 86 k 2024-05-06T14:21:17.041 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 230 k 2024-05-06T14:21:17.041 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:17.102 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-18.2.0-0.el9.x86_64.rpm 1.4 MB/s | 86 kB 00:00 2024-05-06T14:21:17.102 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:17.103 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.3 MB/s | 86 kB 00:00 2024-05-06T14:21:17.103 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:17.110 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:17.110 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:17.173 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:17.173 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:17.316 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:17.413 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:17.749 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:17.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:17.990 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:17.991 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:17.991 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:17.991 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:17.991 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:18.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-05-06T14:21:18.749 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:35 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:18.871 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:18.2.0-0.el9 ceph 3.0 M 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:18.873 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:21:18.874 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:18.874 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.0 M 2024-05-06T14:21:18.874 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 12 M 2024-05-06T14:21:18.874 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:18.984 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-18.2.0-0.el9.x86_64.rpm 27 MB/s | 3.0 MB 00:00 2024-05-06T14:21:18.985 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:18.986 INFO:teuthology.orchestra.run.smithi190.stdout:Total 26 MB/s | 3.0 MB 00:00 2024-05-06T14:21:18.986 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:18.994 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:18.994 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:19.073 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:19.073 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:19.500 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:19.652 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:19.677 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:19.677 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T14:21:19.677 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-06T14:21:19.677 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-06T14:21:19.677 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-06T14:21:19.677 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:20.541 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:20.542 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:20.542 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:20.542 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:20.542 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:20.542 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:20.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-05-06T14:21:21.290 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:38 ago on Mon 06 May 2024 02:19:43 PM UTC. 2024-05-06T14:21:21.417 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:18.2.0-0.el9 ceph 169 k 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-05-06T14:21:21.418 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-05-06T14:21:21.419 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:21.419 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 169 k 2024-05-06T14:21:21.419 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 490 k 2024-05-06T14:21:21.419 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-05-06T14:21:21.488 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-18.2.0-0.el9.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-05-06T14:21:21.489 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:21.489 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.3 MB/s | 169 kB 00:00 2024-05-06T14:21:21.489 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-05-06T14:21:21.497 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-05-06T14:21:21.497 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-05-06T14:21:21.560 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-05-06T14:21:21.560 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-05-06T14:21:21.709 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-05-06T14:21:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:18.2.0-0.el9.x86_64 1/1 2024-05-06T14:21:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:22.431 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-05-06T14:21:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:18.2.0-0.el9.x86_64 2024-05-06T14:21:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:21:22.431 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-05-06T14:21:22.657 DEBUG:teuthology.parallel:result is None 2024-05-06T14:21:22.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-06T14:21:22.657 INFO:teuthology.packaging:ref: None 2024-05-06T14:21:22.657 INFO:teuthology.packaging:tag: v18.2.0 2024-05-06T14:21:22.658 INFO:teuthology.packaging:branch: None 2024-05-06T14:21:22.658 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:21:22.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-05-06T14:21:22.809 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-06T14:21:22.833 INFO:teuthology.orchestra.run.smithi153.stdout:18.2.0-0.el9 2024-05-06T14:21:22.834 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-05-06T14:21:22.834 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-05-06T14:21:22.836 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-05-06T14:21:22.837 INFO:teuthology.packaging:ref: None 2024-05-06T14:21:22.837 INFO:teuthology.packaging:tag: v18.2.0 2024-05-06T14:21:22.837 INFO:teuthology.packaging:branch: None 2024-05-06T14:21:22.837 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:21:22.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dd24139a1eada541a3bc16b6941c5dde975e26d 2024-05-06T14:21:23.084 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-06T14:21:23.108 INFO:teuthology.orchestra.run.smithi190.stdout:18.2.0-0.el9 2024-05-06T14:21:23.108 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-0.el9 2024-05-06T14:21:23.108 INFO:teuthology.task.install:The correct ceph version 18.2.0-0 is installed. 2024-05-06T14:21:23.110 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-06T14:21:23.110 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:21:23.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-06T14:21:23.140 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:21:23.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-06T14:21:23.172 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-06T14:21:23.172 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:21:23.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-05-06T14:21:23.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-06T14:21:23.275 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:21:23.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-05-06T14:21:23.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-06T14:21:23.361 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-06T14:21:23.361 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:21:23.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-06T14:21:23.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-06T14:21:23.452 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:21:23.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-06T14:21:23.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-06T14:21:23.539 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-06T14:21:23.540 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:21:23.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-05-06T14:21:23.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-06T14:21:23.626 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:21:23.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-05-06T14:21:23.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-06T14:21:23.713 INFO:teuthology.run_tasks:Running task print... 2024-05-06T14:21:23.723 INFO:teuthology.task.print:**** done install task... 2024-05-06T14:21:23.724 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-06T14:21:23.830 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.0', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa'} 2024-05-06T14:21:23.830 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.0 2024-05-06T14:21:23.831 INFO:tasks.cephadm:Cluster fsid is eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:21:23.831 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-06T14:21:23.831 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-05-06T14:21:23.831 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi153': '172.21.15.153', 'mon.smithi190': '172.21.15.190'} 2024-05-06T14:21:23.831 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-06T14:21:23.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-05-06T14:21:23.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-05-06T14:21:23.880 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-05-06T14:21:23.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:21:24.025 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55064/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2498-g70ce9539', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2498.g70ce9539'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/70ce9539b62a15b9de429c1f0e91a1e012b099aa/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-03 16:05:34.580410', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/70ce9539b62a15b9de429c1f0e91a1e012b099aa/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-06T14:21:24.276 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref reef, sha1 1758e8ea56486d7ed4a57cf883ac5d1ad36580ad from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-05-06T14:21:24.277 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/reef/1758e8ea56486d7ed4a57cf883ac5d1ad36580ad/centos/9/x86_64/flavors/default/cephadm 2024-05-06T14:21:24.278 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/1758e8ea56486d7ed4a57cf883ac5d1ad36580ad/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-06T14:21:24.699 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 May 6 14:21 /home/ubuntu/cephtest/cephadm 2024-05-06T14:21:24.699 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/reef/1758e8ea56486d7ed4a57cf883ac5d1ad36580ad/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-06T14:21:25.066 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 215333 May 6 14:21 /home/ubuntu/cephtest/cephadm 2024-05-06T14:21:25.067 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-06T14:21:25.083 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-06T14:21:25.111 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.0 on all hosts... 2024-05-06T14:21:25.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-05-06T14:21:25.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 pull 2024-05-06T14:21:26.626 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-05-06T14:21:26.671 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-05-06T14:21:47.494 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-05-06T14:21:47.495 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:21:53.423 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-05-06T14:21:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)", 2024-05-06T14:21:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946", 2024-05-06T14:21:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-05-06T14:21:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007", 2024-05-06T14:21:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "quay.io/ceph/ceph@sha256:d58cf65589d0abf9d5261cd46fb62be3bfb29098febc78c0fcdf116a15274d27" 2024-05-06T14:21:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-05-06T14:21:53.424 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-05-06T14:21:53.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-05-06T14:21:53.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-05-06T14:21:53.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-05-06T14:21:53.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-05-06T14:21:53.563 INFO:tasks.cephadm:Writing seed config... 2024-05-06T14:21:53.564 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-05-06T14:21:53.564 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-05-06T14:21:53.564 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-05-06T14:21:53.564 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-06T14:21:53.565 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-06T14:21:53.566 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-06T14:21:53.567 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-06T14:21:53.568 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:21:53.568 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-06T14:21:53.597 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eacfcfb6-0bb3-11ef-bc97-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-06T14:21:53.597 DEBUG:teuthology.orchestra.run.smithi153:mon.smithi153> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153.service 2024-05-06T14:21:53.640 INFO:tasks.cephadm:Bootstrapping... 2024-05-06T14:21:53.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 -v bootstrap --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.153 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:21:53.798 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-05-06T14:21:53.798 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.0', '-v', 'bootstrap', '--fsid', 'eacfcfb6-0bb3-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.153', '--skip-admin-label'] 2024-05-06T14:21:53.824 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-06T14:21:53.825 INFO:teuthology.orchestra.run.smithi153.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-06T14:21:53.825 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying podman|docker is present... 2024-05-06T14:21:53.850 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-06T14:21:53.850 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying lvm2 is present... 2024-05-06T14:21:53.850 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying time synchronization is in place... 2024-05-06T14:21:53.857 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-06T14:21:53.858 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-06T14:21:53.864 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-06T14:21:53.864 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-05-06T14:21:53.871 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-05-06T14:21:53.878 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-05-06T14:21:53.878 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-05-06T14:21:53.878 INFO:teuthology.orchestra.run.smithi153.stdout:Repeating the final host check... 2024-05-06T14:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-06T14:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-06T14:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl is present 2024-05-06T14:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout:lvcreate is present 2024-05-06T14:21:53.908 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-06T14:21:53.908 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-06T14:21:53.914 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-06T14:21:53.914 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-05-06T14:21:53.921 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-05-06T14:21:53.927 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-05-06T14:21:53.927 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-05-06T14:21:53.927 INFO:teuthology.orchestra.run.smithi153.stdout:Host looks OK 2024-05-06T14:21:53.927 INFO:teuthology.orchestra.run.smithi153.stdout:Cluster fsid: eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:21:53.927 INFO:teuthology.orchestra.run.smithi153.stdout:Acquiring lock 140599241394352 on /run/cephadm/eacfcfb6-0bb3-11ef-bc97-c7b262605968.lock 2024-05-06T14:21:53.928 INFO:teuthology.orchestra.run.smithi153.stdout:Lock 140599241394352 acquired on /run/cephadm/eacfcfb6-0bb3-11ef-bc97-c7b262605968.lock 2024-05-06T14:21:53.928 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 3300 ... 2024-05-06T14:21:53.928 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 6789 ... 2024-05-06T14:21:53.928 INFO:teuthology.orchestra.run.smithi153.stdout:Base mon IP(s) is [172.21.15.153:3300, 172.21.15.153:6789], mon addrv is [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-05-06T14:21:53.931 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.153 metric 100 2024-05-06T14:21:53.931 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.153 metric 100 2024-05-06T14:21:53.935 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-06T14:21:53.935 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-05-06T14:21:53.935 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-05-06T14:21:53.936 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-06T14:21:53.936 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-06T14:21:53.937 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-06T14:21:53.937 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-06T14:21:53.937 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 fe80::c681:6dfa:26e2:6476/64 scope link noprefixroute 2024-05-06T14:21:53.937 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-06T14:21:53.938 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-05-06T14:21:53.938 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-05-06T14:21:53.938 INFO:teuthology.orchestra.run.smithi153.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-06T14:21:53.938 INFO:teuthology.orchestra.run.smithi153.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-06T14:21:53.939 INFO:teuthology.orchestra.run.smithi153.stdout:Pulling container image quay.io/ceph/ceph:v18.2.0... 2024-05-06T14:21:54.703 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-05-06T14:21:54.704 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.0... 2024-05-06T14:21:54.704 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Getting image source signatures 2024-05-06T14:21:54.704 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:3bd20aeff60302f668275dc2005d10679ae56492967a3a5a54fd3dde85333aec 2024-05-06T14:21:54.704 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2024-05-06T14:21:54.704 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying config sha256:dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 2024-05-06T14:21:54.704 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-06T14:21:55.649 INFO:teuthology.orchestra.run.smithi153.stdout:ceph: stdout ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-05-06T14:21:55.649 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 2024-05-06T14:21:55.649 INFO:teuthology.orchestra.run.smithi153.stdout:Extracting ceph user uid/gid from container image... 2024-05-06T14:21:56.579 INFO:teuthology.orchestra.run.smithi153.stdout:stat: stdout 167 167 2024-05-06T14:21:56.579 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial keys... 2024-05-06T14:21:57.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQCE5zhm/UoWNhAA1ZcJLwH/Hf0py7T82dJY3A== 2024-05-06T14:22:06.990 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQCI5zhmqOlDGBAA+JvzC3JV8M0uxIV58NkcGA== 2024-05-06T14:22:09.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQCQ5zhm7HvgOhAABu2smOMuWxcNtSeTpHygSg== 2024-05-06T14:22:09.555 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial monmap... 2024-05-06T14:22:10.604 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-06T14:22:10.604 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-05-06T14:22:10.604 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:10.604 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-06T14:22:10.605 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool for smithi153 [v2:172.21.15.153:3300,v1:172.21.15.153:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-06T14:22:10.605 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = pacific 2024-05-06T14:22:10.605 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: set fsid to eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:10.605 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-06T14:22:10.605 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:22:10.605 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mon... 2024-05-06T14:22:11.516 INFO:teuthology.orchestra.run.smithi153.stdout:create mon.smithi153 on 2024-05-06T14:22:11.733 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-06T14:22:11.870 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-06T14:22:12.063 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968.target. 2024-05-06T14:22:12.063 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968.target. 2024-05-06T14:22:12.276 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153 2024-05-06T14:22:12.277 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153.service: Unit ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153.service not loaded. 2024-05-06T14:22:12.437 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968.target.wants/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153.service → /etc/systemd/system/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@.service. 2024-05-06T14:22:12.489 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 systemd[1]: Starting Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:22:12.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 podman[26783]: 2024-05-06 14:22:12.694501192 +0000 UTC m=+0.114994462 container create 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-06T14:22:12.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 podman[26783]: 2024-05-06 14:22:12.597602071 +0000 UTC m=+0.018095369 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-05-06T14:22:12.948 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-06T14:22:12.948 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-05-06T14:22:12.954 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-06T14:22:12.954 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-05-06T14:22:12.955 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-05-06T14:22:12.955 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-05-06T14:22:12.955 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon to start... 2024-05-06T14:22:12.955 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon... 2024-05-06T14:22:13.079 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 podman[26783]: 2024-05-06 14:22:12.863265008 +0000 UTC m=+0.283758286 container init 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-06T14:22:13.079 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 podman[26783]: 2024-05-06 14:22:12.86624112 +0000 UTC m=+0.286734393 container start 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:22:13.080 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 ceph-mon[26802]: mkfs eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:13.080 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 ceph-mon[26802]: mon.smithi153 is new leader, mons smithi153 in quorum (ranks 0) 2024-05-06T14:22:13.080 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 bash[26783]: 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 2024-05-06T14:22:13.080 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:12 smithi153 systemd[1]: Started Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968. 2024-05-06T14:22:14.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:13 smithi153 ceph-mon[26802]: mon.smithi153 is new leader, mons smithi153 in quorum (ranks 0) 2024-05-06T14:22:14.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:13 smithi153 ceph-mon[26802]: monmap e1: 1 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} 2024-05-06T14:22:14.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:13 smithi153 ceph-mon[26802]: fsmap 2024-05-06T14:22:14.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:13 smithi153 ceph-mon[26802]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T14:22:14.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:13 smithi153 ceph-mon[26802]: mgrmap e1: no daemons active 2024-05-06T14:22:14.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:13 smithi153 ceph-mon[26802]: from='client.? 172.21.15.153:0/12035518' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout cluster: 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout id: eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout services: 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi153 (age 0.569029s) 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout data: 2024-05-06T14:22:14.741 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pgs: 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.396+0000 7f340847a700 1 Processor -- start 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.396+0000 7f340847a700 1 -- start start 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.396+0000 7f340847a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.396+0000 7f340847a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3400106460 con 0x7f3400105b10 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.397+0000 7f3406216700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:14.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.397+0000 7f3406216700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39946/0 (socket says 172.21.15.153:39946) 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.397+0000 7f3406216700 1 -- 172.21.15.153:0/3062858557 learned_addr learned my addr 172.21.15.153:0/3062858557 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.397+0000 7f3406216700 1 -- 172.21.15.153:0/3062858557 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34001065a0 con 0x7f3400105b10 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.397+0000 7f3406216700 1 --2- 172.21.15.153:0/3062858557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f33f00098a0 tx=0x7f33f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4eb0c727352e6f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f3405214700 1 -- 172.21.15.153:0/3062858557 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33f0004180 con 0x7f3400105b10 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f3405214700 1 -- 172.21.15.153:0/3062858557 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f00042e0 con 0x7f3400105b10 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f3405214700 1 -- 172.21.15.153:0/3062858557 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33f00036a0 con 0x7f3400105b10 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 -- 172.21.15.153:0/3062858557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 msgr2=0x7f3400105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 --2- 172.21.15.153:0/3062858557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400105f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f33f00098a0 tx=0x7f33f0009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 -- 172.21.15.153:0/3062858557 shutdown_connections 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 --2- 172.21.15.153:0/3062858557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400105f20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 -- 172.21.15.153:0/3062858557 >> 172.21.15.153:0/3062858557 conn(0x7f3400101180 msgr2=0x7f3400103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:14.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 -- 172.21.15.153:0/3062858557 shutdown_connections 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.398+0000 7f340847a700 1 -- 172.21.15.153:0/3062858557 wait complete. 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f340847a700 1 Processor -- start 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f340847a700 1 -- start start 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f340847a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f340847a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3400199560 con 0x7f3400105b10 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f3406216700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f3406216700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39954/0 (socket says 172.21.15.153:39954) 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f3406216700 1 -- 172.21.15.153:0/12035518 learned_addr learned my addr 172.21.15.153:0/12035518 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.399+0000 7f3406216700 1 -- 172.21.15.153:0/12035518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0009550 con 0x7f3400105b10 2024-05-06T14:22:14.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f3406216700 1 --2- 172.21.15.153:0/12035518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400199020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3400106840 tx=0x7f33f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33f0003ce0 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f0020070 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3400199760 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33f0011630 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3400199b80 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f33f0022030 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.400+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f33f0025d00 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.401+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3400192890 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.403+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f33f0025480 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.472+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f340002cd90 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f33f77fe700 1 -- 172.21.15.153:0/12035518 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f33f0011bd0 con 0x7f3400105b10 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 msgr2=0x7f3400199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:14.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 --2- 172.21.15.153:0/12035518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400199020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3400106840 tx=0x7f33f0004750 comp rx=0 tx=0).stop 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 shutdown_connections 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 --2- 172.21.15.153:0/12035518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3400105b10 0x7f3400199020 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 >> 172.21.15.153:0/12035518 conn(0x7f3400101180 msgr2=0x7f340018d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 shutdown_connections 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:13.473+0000 7f340847a700 1 -- 172.21.15.153:0/12035518 wait complete. 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:mon is available 2024-05-06T14:22:14.746 INFO:teuthology.orchestra.run.smithi153.stdout:Assimilating anything we can from ceph.conf... 2024-05-06T14:22:15.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:15 smithi153 ceph-mon[26802]: from='client.? 172.21.15.153:0/540311939' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T14:22:15.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:15 smithi153 ceph-mon[26802]: from='client.? 172.21.15.153:0/540311939' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T14:22:15.821 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:15.821 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-05-06T14:22:15.821 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-06T14:22:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.184+0000 7fd5d57d5700 1 Processor -- start 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.184+0000 7fd5d57d5700 1 -- start start 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5d57d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0104e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5d57d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d0105340 con 0x7fd5d01049f0 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5ceffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5ceffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0104e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39962/0 (socket says 172.21.15.153:39962) 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5ceffd700 1 -- 172.21.15.153:0/3154660950 learned_addr learned my addr 172.21.15.153:0/3154660950 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5ceffd700 1 -- 172.21.15.153:0/3154660950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0105480 con 0x7fd5d01049f0 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5ceffd700 1 --2- 172.21.15.153:0/3154660950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0104e00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd5c00098a0 tx=0x7fd5c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b1d6687e20bdb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5cdffb700 1 -- 172.21.15.153:0/3154660950 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5c0004180 con 0x7fd5d01049f0 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5cdffb700 1 -- 172.21.15.153:0/3154660950 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c00042e0 con 0x7fd5d01049f0 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.185+0000 7fd5cdffb700 1 -- 172.21.15.153:0/3154660950 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5c00036a0 con 0x7fd5d01049f0 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 -- 172.21.15.153:0/3154660950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 msgr2=0x7fd5d0104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 --2- 172.21.15.153:0/3154660950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0104e00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd5c00098a0 tx=0x7fd5c0009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 -- 172.21.15.153:0/3154660950 shutdown_connections 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 --2- 172.21.15.153:0/3154660950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0104e00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 -- 172.21.15.153:0/3154660950 >> 172.21.15.153:0/3154660950 conn(0x7fd5d0100080 msgr2=0x7fd5d0102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 -- 172.21.15.153:0/3154660950 shutdown_connections 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.186+0000 7fd5d57d5700 1 -- 172.21.15.153:0/3154660950 wait complete. 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5d57d5700 1 Processor -- start 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5d57d5700 1 -- start start 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5d57d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5d57d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d0198430 con 0x7fd5d01049f0 2024-05-06T14:22:15.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5ceffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5ceffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39964/0 (socket says 172.21.15.153:39964) 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5ceffd700 1 -- 172.21.15.153:0/540311939 learned_addr learned my addr 172.21.15.153:0/540311939 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5ceffd700 1 -- 172.21.15.153:0/540311939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c0009550 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.187+0000 7fd5ceffd700 1 --2- 172.21.15.153:0/540311939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0197ef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0105720 tx=0x7fd5c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5c0003ce0 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0020070 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d0198630 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5c0011630 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d0198a50 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd5c0022030 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.188+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd5c0025d00 con 0x7fd5d01049f0 2024-05-06T14:22:15.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.189+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d0105db0 con 0x7fd5d01049f0 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.191+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fd5c0025480 con 0x7fd5d01049f0 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.258+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd5d0105f70 con 0x7fd5d01049f0 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.261+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd5c0011bd0 con 0x7fd5d01049f0 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.261+0000 7fd5b7fff700 1 -- 172.21.15.153:0/540311939 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fd5c001c560 con 0x7fd5d01049f0 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 msgr2=0x7fd5d0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 --2- 172.21.15.153:0/540311939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0197ef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0105720 tx=0x7fd5c0004750 comp rx=0 tx=0).stop 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 shutdown_connections 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 --2- 172.21.15.153:0/540311939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5d01049f0 0x7fd5d0197ef0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 >> 172.21.15.153:0/540311939 conn(0x7fd5d0100080 msgr2=0x7fd5d018c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 shutdown_connections 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:15.262+0000 7fd5d57d5700 1 -- 172.21.15.153:0/540311939 wait complete. 2024-05-06T14:22:15.826 INFO:teuthology.orchestra.run.smithi153.stdout:Generating new minimal ceph.conf... 2024-05-06T14:22:16.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:16 smithi153 ceph-mon[26802]: from='client.? 172.21.15.153:0/138999861' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:22:16.902 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f8341711700 1 Processor -- start 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f8341711700 1 -- start start 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f8341711700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c1045a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f8341711700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c104ae0 con 0x7f833c104190 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f833affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c1045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f833affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c1045a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39978/0 (socket says 172.21.15.153:39978) 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f833affd700 1 -- 172.21.15.153:0/2382042345 learned_addr learned my addr 172.21.15.153:0/2382042345 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.240+0000 7f833affd700 1 -- 172.21.15.153:0/2382042345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c104c20 con 0x7f833c104190 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f833affd700 1 --2- 172.21.15.153:0/2382042345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c1045a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f832c0098a0 tx=0x7f832c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b19474c24ccda31 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8339ffb700 1 -- 172.21.15.153:0/2382042345 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c004180 con 0x7f833c104190 2024-05-06T14:22:16.903 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8339ffb700 1 -- 172.21.15.153:0/2382042345 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f832c0042e0 con 0x7f833c104190 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8339ffb700 1 -- 172.21.15.153:0/2382042345 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c011a50 con 0x7f833c104190 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 -- 172.21.15.153:0/2382042345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 msgr2=0x7f833c1045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 --2- 172.21.15.153:0/2382042345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c1045a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f832c0098a0 tx=0x7f832c009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 -- 172.21.15.153:0/2382042345 shutdown_connections 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 --2- 172.21.15.153:0/2382042345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c1045a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 -- 172.21.15.153:0/2382042345 >> 172.21.15.153:0/2382042345 conn(0x7f833c0ff7e0 msgr2=0x7f833c101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 -- 172.21.15.153:0/2382042345 shutdown_connections 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.241+0000 7f8341711700 1 -- 172.21.15.153:0/2382042345 wait complete. 2024-05-06T14:22:16.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.242+0000 7f8341711700 1 Processor -- start 2024-05-06T14:22:16.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.242+0000 7f8341711700 1 -- start start 2024-05-06T14:22:16.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.242+0000 7f8341711700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:16.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.242+0000 7f8341711700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c199d70 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f833affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f833affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39988/0 (socket says 172.21.15.153:39988) 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f833affd700 1 -- 172.21.15.153:0/138999861 learned_addr learned my addr 172.21.15.153:0/138999861 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f833affd700 1 -- 172.21.15.153:0/138999861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c009550 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f833affd700 1 --2- 172.21.15.153:0/138999861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c199830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f832c000c00 tx=0x7f832c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c003b90 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f832c0248b0 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c199f70 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c01a8e0 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c19a390 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f832c02d3f0 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.243+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f832c0194a0 con 0x7f833c104190 2024-05-06T14:22:16.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.244+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c193040 con 0x7f833c104190 2024-05-06T14:22:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.246+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f832c024db0 con 0x7f833c104190 2024-05-06T14:22:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.313+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f833c02cc30 con 0x7f833c104190 2024-05-06T14:22:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.313+0000 7f8323fff700 1 -- 172.21.15.153:0/138999861 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f832c024ad0 con 0x7f833c104190 2024-05-06T14:22:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 msgr2=0x7f833c199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 --2- 172.21.15.153:0/138999861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c199830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f832c000c00 tx=0x7f832c011ab0 comp rx=0 tx=0).stop 2024-05-06T14:22:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 shutdown_connections 2024-05-06T14:22:16.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 --2- 172.21.15.153:0/138999861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f833c104190 0x7f833c199830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:16.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 >> 172.21.15.153:0/138999861 conn(0x7f833c0ff7e0 msgr2=0x7f833c18de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:16.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 shutdown_connections 2024-05-06T14:22:16.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:16.314+0000 7f8341711700 1 -- 172.21.15.153:0/138999861 wait complete. 2024-05-06T14:22:16.908 INFO:teuthology.orchestra.run.smithi153.stdout:Restarting the monitor... 2024-05-06T14:22:17.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:16 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:22:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:17 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153[26798]: 2024-05-06T14:22:17.183+0000 7f921498c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:22:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:17 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153[26798]: 2024-05-06T14:22:17.183+0000 7f921498c700 -1 mon.smithi153@0(leader) e1 *** Got Signal Terminated *** 2024-05-06T14:22:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:17 smithi153 podman[27048]: 2024-05-06 14:22:17.297069043 +0000 UTC m=+0.229237554 container died 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:22:18.390 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 podman[27048]: 2024-05-06 14:22:18.200871221 +0000 UTC m=+1.133039731 container cleanup 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2) 2024-05-06T14:22:18.390 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 bash[27048]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153 2024-05-06T14:22:18.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 podman[27062]: 2024-05-06 14:22:18.401830593 +0000 UTC m=+1.099788834 container remove 45d06041fd691930e1f1717cc39757c1f8c8281b65a0438a3cdca8023205d3a7 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD) 2024-05-06T14:22:18.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 systemd[1]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153.service: Deactivated successfully. 2024-05-06T14:22:18.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968. 2024-05-06T14:22:18.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 systemd[1]: Starting Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:22:18.973 INFO:teuthology.orchestra.run.smithi153.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-06T14:22:18.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 podman[27158]: 2024-05-06 14:22:18.637282731 +0000 UTC m=+0.018813328 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph:v18.2.0 2024-05-06T14:22:18.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 podman[27158]: 2024-05-06 14:22:18.736070488 +0000 UTC m=+0.117601111 container create 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, ceph=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:22:18.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 podman[27158]: 2024-05-06 14:22:18.905096486 +0000 UTC m=+0.286627067 container init 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 podman[27158]: 2024-05-06 14:22:18.90786921 +0000 UTC m=+0.289399795 container start 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, ceph=True, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mon, pid 2 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: pidfile_write: ignore empty --pid-file 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: load: jerasure load: lrc 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: RocksDB version: 7.9.2 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Git sha 0 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Compile date 2023-08-03 19:21:13 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: DB SUMMARY 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: DB Session ID: SM1N688T8HMGNVR9YDBZ 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: CURRENT file: CURRENT 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-06T14:22:18.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 1, files: 000008.sst 2024-05-06T14:22:18.987 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000009.log size: 98270 ; 2024-05-06T14:22:18.987 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.error_if_exists: 0 2024-05-06T14:22:18.987 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.create_if_missing: 0 2024-05-06T14:22:18.987 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.paranoid_checks: 1 2024-05-06T14:22:18.987 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T14:22:18.987 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.env: 0x559a4337d720 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.info_log: 0x559a463192c0 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.statistics: (nil) 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.use_fsync: 0 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_log_file_size: 0 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.allow_fallocate: 1 2024-05-06T14:22:18.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.use_direct_reads: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.db_log_dir: 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.wal_dir: 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T14:22:18.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.write_buffer_manager: 0x559a4558a3c0 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.unordered_write: 0 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T14:22:18.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.row_cache: None 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.wal_filter: None 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.two_write_queues: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.wal_compression: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.atomic_flush: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.log_readahead_size: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T14:22:18.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_background_jobs: 2 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_background_compactions: -1 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_subcompactions: 1 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T14:22:18.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_open_files: -1 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_background_flushes: -1 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Compression algorithms supported: 2024-05-06T14:22:18.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kZSTD supported: 0 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kXpressCompression supported: 0 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kZlibCompression supported: 1 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kSnappyCompression supported: 1 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kLZ4Compression supported: 1 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: kBZip2Compression supported: 0 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000010 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.merge_operator: 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_filter: None 2024-05-06T14:22:18.994 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a463193c0) 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 4 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-05-06T14:22:18.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x559a4562b1f0 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-05-06T14:22:18.996 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-05-06T14:22:18.997 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: bloomfilter 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 5 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_auto_readahead_size: 262144 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: prepopulate_block_cache: 0 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: initial_auto_readahead_size: 8192 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T14:22:18.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression: NoCompression 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.num_levels: 7 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T14:22:18.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T14:22:19.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T14:22:19.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.table_properties_collectors: 2024-05-06T14:22:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.inplace_update_support: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.bloom_locality: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.max_successive_merges: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.ttl: 2592000 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T14:22:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.enable_blob_files: false 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.min_blob_size: 0 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7a10fbeb-b2ae-446b-bb3b-8dfd106d6f83 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715005338936282, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-06T14:22:19.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715005338937314, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93521, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91543, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84152, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715005338, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7a10fbeb-b2ae-446b-bb3b-8dfd106d6f83", "db_session_id": "SM1N688T8HMGNVR9YDBZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715005338937395, "job": 1, "event": "recovery_finished"} 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559a456c8000 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: DB pointer 0x559a456b4000 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T14:22:19.005 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 93.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 93.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T14:22:19.006 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 29.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 29.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache BinnedLRUCache@0x559a4562b1f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-05-06T14:22:19.007 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: starting mon.smithi153 rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???) e1 preinit fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).mds e1 new map 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).mds e1 print_map 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e1 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: -1 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: No filesystems configured 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T14:22:19.008 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expand map: {default=false} 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta from 'false' to 'false' 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expanded map: {default=false} 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expand map: {default=info} 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta from 'info' to 'info' 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expanded map: {default=info} 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expand map: {default=daemon} 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta from 'daemon' to 'daemon' 2024-05-06T14:22:19.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expanded map: {default=daemon} 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expand map: {default=debug} 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta from 'debug' to 'debug' 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: expand_channel_meta expanded map: {default=debug} 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 bash[27158]: 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 systemd[1]: Started Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968. 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mon.smithi153 is new leader, mons smithi153 in quorum (ranks 0) 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: monmap e1: 1 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: fsmap 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T14:22:19.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:18 smithi153 ceph-mon[27177]: mgrmap e1: no daemons active 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.416+0000 7f606a226700 1 Processor -- start 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.416+0000 7f606a226700 1 -- start start 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f606a226700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f6064108a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f606a226700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60640744d0 con 0x7f6064106650 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f60637fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f6064108a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f60637fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f6064108a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39828/0 (socket says 172.21.15.153:39828) 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f60637fe700 1 -- 172.21.15.153:0/1894718873 learned_addr learned my addr 172.21.15.153:0/1894718873 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f60637fe700 1 -- 172.21.15.153:0/1894718873 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064074610 con 0x7f6064106650 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.417+0000 7f60637fe700 1 --2- 172.21.15.153:0/1894718873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f6064108a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f60540098a0 tx=0x7f6054009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84272d2abf7220e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:20.131 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f6062ffd700 1 -- 172.21.15.153:0/1894718873 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054004180 con 0x7f6064106650 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f6062ffd700 1 -- 172.21.15.153:0/1894718873 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f60540042e0 con 0x7f6064106650 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f6062ffd700 1 -- 172.21.15.153:0/1894718873 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054011a50 con 0x7f6064106650 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 -- 172.21.15.153:0/1894718873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 msgr2=0x7f6064108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 --2- 172.21.15.153:0/1894718873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f6064108a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f60540098a0 tx=0x7f6054009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 -- 172.21.15.153:0/1894718873 shutdown_connections 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 --2- 172.21.15.153:0/1894718873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f6064108a30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 -- 172.21.15.153:0/1894718873 >> 172.21.15.153:0/1894718873 conn(0x7f6064100070 msgr2=0x7f6064102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 -- 172.21.15.153:0/1894718873 shutdown_connections 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.418+0000 7f606a226700 1 -- 172.21.15.153:0/1894718873 wait complete. 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f606a226700 1 Processor -- start 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f606a226700 1 -- start start 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f606a226700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f60641997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f606a226700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064199d20 con 0x7f6064106650 2024-05-06T14:22:20.132 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f60637fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f60641997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f60637fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f60641997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39844/0 (socket says 172.21.15.153:39844) 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f60637fe700 1 -- 172.21.15.153:0/1603246708 learned_addr learned my addr 172.21.15.153:0/1603246708 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.419+0000 7f60637fe700 1 -- 172.21.15.153:0/1603246708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054009550 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f60637fe700 1 --2- 172.21.15.153:0/1603246708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f60641997e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6054006af0 tx=0x7f6054011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054003b90 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6064199f80 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f605401a8a0 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054019880 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606419a400 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f605402d440 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.420+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6054024810 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.421+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6064109b30 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.423+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f605401aa10 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.490+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6064109d20 con 0x7f6064106650 2024-05-06T14:22:20.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.493+0000 7f60617fa700 1 -- 172.21.15.153:0/1603246708 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f605401abb0 con 0x7f6064106650 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 msgr2=0x7f60641997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 --2- 172.21.15.153:0/1603246708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f60641997e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6054006af0 tx=0x7f6054011a90 comp rx=0 tx=0).stop 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 shutdown_connections 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 --2- 172.21.15.153:0/1603246708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6064106650 0x7f60641997e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 >> 172.21.15.153:0/1603246708 conn(0x7f6064100070 msgr2=0x7f6064101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 shutdown_connections 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:19.494+0000 7f606a226700 1 -- 172.21.15.153:0/1603246708 wait complete. 2024-05-06T14:22:20.134 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-06T14:22:20.148 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:22:20.148 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mgr... 2024-05-06T14:22:20.148 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-06T14:22:20.149 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-06T14:22:20.149 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying port 0.0.0.0:8443 ... 2024-05-06T14:22:20.387 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mgr.smithi153.dolfed 2024-05-06T14:22:20.387 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mgr.smithi153.dolfed.service: Unit ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mgr.smithi153.dolfed.service not loaded. 2024-05-06T14:22:20.536 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968.target.wants/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mgr.smithi153.dolfed.service → /etc/systemd/system/ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@.service. 2024-05-06T14:22:20.774 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:20 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1603246708' entity='client.admin' 2024-05-06T14:22:21.083 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-06T14:22:21.083 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-05-06T14:22:21.090 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-06T14:22:21.091 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-05-06T14:22:21.091 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-05-06T14:22:21.091 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-05-06T14:22:21.099 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-06T14:22:21.099 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-05-06T14:22:21.106 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-06T14:22:21.106 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-05-06T14:22:21.106 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-05-06T14:22:21.106 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-05-06T14:22:21.106 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr to start... 2024-05-06T14:22:21.106 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr... 2024-05-06T14:22:22.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:21 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2234299129' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T14:22:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:22:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "eacfcfb6-0bb3-11ef-bc97-c7b262605968", 2024-05-06T14:22:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-05-06T14:22:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-06T14:22:22.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-06T14:22:22.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-06T14:22:22.276 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.276 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-06T14:22:22.276 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:22.276 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-06T14:22:22.276 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-06T14:22:22.276 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-06T14:22:22.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-06T14:22:22.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-05-06T14:22:12.905905+0000", 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:22:22.286 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.608+0000 7ff4e9dc3700 1 Processor -- start 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.608+0000 7ff4e9dc3700 1 -- start start 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e9dc3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e41089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e9dc3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4074510 con 0x7ff4e4106610 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e41089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e41089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39874/0 (socket says 172.21.15.153:39874) 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e37fe700 1 -- 172.21.15.153:0/3497620510 learned_addr learned my addr 172.21.15.153:0/3497620510 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e37fe700 1 -- 172.21.15.153:0/3497620510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4074650 con 0x7ff4e4106610 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.609+0000 7ff4e37fe700 1 --2- 172.21.15.153:0/3497620510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e41089f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff4d40098a0 tx=0x7ff4d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7eb72bc65c90df5 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e27fc700 1 -- 172.21.15.153:0/3497620510 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4d4004180 con 0x7ff4e4106610 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e27fc700 1 -- 172.21.15.153:0/3497620510 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff4d40042e0 con 0x7ff4e4106610 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e27fc700 1 -- 172.21.15.153:0/3497620510 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4d4011a50 con 0x7ff4e4106610 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/3497620510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 msgr2=0x7ff4e41089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 --2- 172.21.15.153:0/3497620510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e41089f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff4d40098a0 tx=0x7ff4d4009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:22.287 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/3497620510 shutdown_connections 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 --2- 172.21.15.153:0/3497620510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e41089f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/3497620510 >> 172.21.15.153:0/3497620510 conn(0x7ff4e4100070 msgr2=0x7ff4e4102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/3497620510 shutdown_connections 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.610+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/3497620510 wait complete. 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e9dc3700 1 Processor -- start 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e9dc3700 1 -- start start 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e9dc3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e4199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e9dc3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4199d90 con 0x7ff4e4106610 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e4199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e4199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39890/0 (socket says 172.21.15.153:39890) 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.611+0000 7ff4e37fe700 1 -- 172.21.15.153:0/2234299129 learned_addr learned my addr 172.21.15.153:0/2234299129 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e37fe700 1 -- 172.21.15.153:0/2234299129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4009550 con 0x7ff4e4106610 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e37fe700 1 --2- 172.21.15.153:0/2234299129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e4199850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000c00 tx=0x7ff4d4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:22.288 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4d4003b90 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e4199f90 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff4d40248b0 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4d401a8e0 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e419a3b0 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff4d402d3f0 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.612+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff4d4019490 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.613+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e4062380 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.617+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7ff4d4024a20 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.690+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff4e419cee0 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.691+0000 7ff4e0ff9700 1 -- 172.21.15.153:0/2234299129 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7ff4d4035090 con 0x7ff4e4106610 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.692+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 msgr2=0x7ff4e4199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.692+0000 7ff4e9dc3700 1 --2- 172.21.15.153:0/2234299129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e4199850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000c00 tx=0x7ff4d4011ab0 comp rx=0 tx=0).stop 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.692+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 shutdown_connections 2024-05-06T14:22:22.289 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.692+0000 7ff4e9dc3700 1 --2- 172.21.15.153:0/2234299129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4e4106610 0x7ff4e4199850 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:22.290 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.692+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 >> 172.21.15.153:0/2234299129 conn(0x7ff4e4100070 msgr2=0x7ff4e4101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:22.290 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.693+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 shutdown_connections 2024-05-06T14:22:22.290 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:21.693+0000 7ff4e9dc3700 1 -- 172.21.15.153:0/2234299129 wait complete. 2024-05-06T14:22:22.290 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (1/15)... 2024-05-06T14:22:25.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:24 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/815915743' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "eacfcfb6-0bb3-11ef-bc97-c7b262605968", 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-06T14:22:25.428 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-06T14:22:25.429 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-06T14:22:25.430 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:25.431 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-05-06T14:22:12.905905+0000", 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.752+0000 7f6434bce700 1 Processor -- start 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.752+0000 7f6434bce700 1 -- start start 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.752+0000 7f6434bce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430108a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.752+0000 7f6434bce700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430079d30 con 0x7f6430106680 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430108a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430108a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39904/0 (socket says 172.21.15.153:39904) 2024-05-06T14:22:25.432 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642e59c700 1 -- 172.21.15.153:0/3119252920 learned_addr learned my addr 172.21.15.153:0/3119252920 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642e59c700 1 -- 172.21.15.153:0/3119252920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6430079e70 con 0x7f6430106680 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642e59c700 1 --2- 172.21.15.153:0/3119252920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430108a60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64200098a0 tx=0x7f6420009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d58b8ffa25242d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642d59a700 1 -- 172.21.15.153:0/3119252920 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6420004180 con 0x7f6430106680 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642d59a700 1 -- 172.21.15.153:0/3119252920 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f64200042e0 con 0x7f6430106680 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.753+0000 7f642d59a700 1 -- 172.21.15.153:0/3119252920 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6420011a50 con 0x7f6430106680 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 -- 172.21.15.153:0/3119252920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 msgr2=0x7f6430108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 --2- 172.21.15.153:0/3119252920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430108a60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64200098a0 tx=0x7f6420009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 -- 172.21.15.153:0/3119252920 shutdown_connections 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 --2- 172.21.15.153:0/3119252920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430108a60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 -- 172.21.15.153:0/3119252920 >> 172.21.15.153:0/3119252920 conn(0x7f6430100060 msgr2=0x7f64301024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 -- 172.21.15.153:0/3119252920 shutdown_connections 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.754+0000 7f6434bce700 1 -- 172.21.15.153:0/3119252920 wait complete. 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f6434bce700 1 Processor -- start 2024-05-06T14:22:25.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f6434bce700 1 -- start start 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f6434bce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f6434bce700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430199da0 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f642e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f642e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39918/0 (socket says 172.21.15.153:39918) 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f642e59c700 1 -- 172.21.15.153:0/815915743 learned_addr learned my addr 172.21.15.153:0/815915743 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f642e59c700 1 -- 172.21.15.153:0/815915743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420009550 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.755+0000 7f642e59c700 1 --2- 172.21.15.153:0/815915743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430199860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6420003de0 tx=0x7f6420011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6420003930 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643019a000 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6420003a90 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6420019690 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643019a480 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6420019c50 con 0x7f6430106680 2024-05-06T14:22:25.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.756+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f642002c3a0 con 0x7f6430106680 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.757+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6430062380 con 0x7f6430106680 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.760+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f6420042b00 con 0x7f6430106680 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.832+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f643019ce60 con 0x7f6430106680 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.832+0000 7f641f7fe700 1 -- 172.21.15.153:0/815915743 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f6420031030 con 0x7f6430106680 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 msgr2=0x7f6430199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 --2- 172.21.15.153:0/815915743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430199860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6420003de0 tx=0x7f6420011ab0 comp rx=0 tx=0).stop 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 shutdown_connections 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 --2- 172.21.15.153:0/815915743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6430106680 0x7f6430199860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 >> 172.21.15.153:0/815915743 conn(0x7f6430100060 msgr2=0x7f64301017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 shutdown_connections 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:24.833+0000 7f6434bce700 1 -- 172.21.15.153:0/815915743 wait complete. 2024-05-06T14:22:25.435 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (2/15)... 2024-05-06T14:22:27.953 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: Activating manager daemon smithi153.dolfed 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: mgrmap e2: smithi153.dolfed(active, starting, since 0.00290186s) 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr metadata", "who": "smithi153.dolfed", "id": "smithi153.dolfed"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: Manager daemon smithi153.dolfed is now available 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/mirror_snapshot_schedule"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/trash_purge_schedule"}]: dispatch 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:27.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:27.955 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:27 smithi153 ceph-mon[27177]: from='mgr.14100 172.21.15.153:0/2600031226' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:28.567 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:28.567 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:22:28.567 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "eacfcfb6-0bb3-11ef-bc97-c7b262605968", 2024-05-06T14:22:28.567 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-05-06T14:22:28.567 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:28.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-06T14:22:28.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-06T14:22:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-05-06T14:22:12.905905+0000", 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:22:28.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57de650700 1 Processor -- start 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57de650700 1 -- start start 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57de650700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57de650700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d8106460 con 0x7f57d8105b10 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57d7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57d7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40002/0 (socket says 172.21.15.153:40002) 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57d7fff700 1 -- 172.21.15.153:0/1006494458 learned_addr learned my addr 172.21.15.153:0/1006494458 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.876+0000 7f57d7fff700 1 -- 172.21.15.153:0/1006494458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d81065a0 con 0x7f57d8105b10 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57d7fff700 1 --2- 172.21.15.153:0/1006494458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d8105f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f57c80098a0 tx=0x7f57c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab494139ff966d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57d6ffd700 1 -- 172.21.15.153:0/1006494458 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c8004180 con 0x7f57d8105b10 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57d6ffd700 1 -- 172.21.15.153:0/1006494458 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f57c80042e0 con 0x7f57d8105b10 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57d6ffd700 1 -- 172.21.15.153:0/1006494458 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c8011a50 con 0x7f57d8105b10 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57de650700 1 -- 172.21.15.153:0/1006494458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 msgr2=0x7f57d8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57de650700 1 --2- 172.21.15.153:0/1006494458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d8105f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f57c80098a0 tx=0x7f57c8009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:28.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57de650700 1 -- 172.21.15.153:0/1006494458 shutdown_connections 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57de650700 1 --2- 172.21.15.153:0/1006494458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d8105f20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.877+0000 7f57de650700 1 -- 172.21.15.153:0/1006494458 >> 172.21.15.153:0/1006494458 conn(0x7f57d8101180 msgr2=0x7f57d8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57de650700 1 -- 172.21.15.153:0/1006494458 shutdown_connections 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57de650700 1 -- 172.21.15.153:0/1006494458 wait complete. 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57de650700 1 Processor -- start 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57de650700 1 -- start start 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57de650700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d807bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57de650700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d807c180 con 0x7f57d8105b10 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57d7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d807bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57d7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d807bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40008/0 (socket says 172.21.15.153:40008) 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.878+0000 7f57d7fff700 1 -- 172.21.15.153:0/4108021966 learned_addr learned my addr 172.21.15.153:0/4108021966 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57d7fff700 1 -- 172.21.15.153:0/4108021966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c8009550 con 0x7f57d8105b10 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57d7fff700 1 --2- 172.21.15.153:0/4108021966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d807bc40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f57c80119e0 tx=0x7f57c8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:28.573 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c8003b90 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f57c801a8a0 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d807a290 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c8019930 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.879+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d807a6b0 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.880+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 2) v1 ==== 44840+0+0 (secure 0 0 0) 0x7f57c8019bf0 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.880+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d8062380 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.880+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f57c804c540 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.882+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f57c802d440 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.952+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f57d807b130 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.953+0000 7f57d57fa700 1 -- 172.21.15.153:0/4108021966 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f57c802b340 con 0x7f57d8105b10 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.953+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 msgr2=0x7f57d807bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.953+0000 7f57de650700 1 --2- 172.21.15.153:0/4108021966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d807bc40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f57c80119e0 tx=0x7f57c8011ac0 comp rx=0 tx=0).stop 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.953+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 shutdown_connections 2024-05-06T14:22:28.574 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.953+0000 7f57de650700 1 --2- 172.21.15.153:0/4108021966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57d8105b10 0x7f57d807bc40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:28.575 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.954+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 >> 172.21.15.153:0/4108021966 conn(0x7f57d8101180 msgr2=0x7f57d8101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:28.575 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.954+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 shutdown_connections 2024-05-06T14:22:28.575 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:27.954+0000 7f57de650700 1 -- 172.21.15.153:0/4108021966 wait complete. 2024-05-06T14:22:28.575 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (3/15)... 2024-05-06T14:22:29.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:28 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/4108021966' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T14:22:29.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:28 smithi153 ceph-mon[27177]: mgrmap e3: smithi153.dolfed(active, since 1.00651s) 2024-05-06T14:22:31.240 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:30 smithi153 ceph-mon[27177]: mgrmap e4: smithi153.dolfed(active, since 2s) 2024-05-06T14:22:31.767 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:31.767 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "eacfcfb6-0bb3-11ef-bc97-c7b262605968", 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-06T14:22:31.768 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-06T14:22:31.769 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-06T14:22:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-06T14:22:31.771 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-06T14:22:31.772 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-05-06T14:22:12.905905+0000", 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.979+0000 7f28fe4aa700 1 Processor -- start 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.979+0000 7f28fe4aa700 1 -- start start 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.979+0000 7f28fe4aa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28fe4aa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f8106460 con 0x7f28f8105b10 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60392/0 (socket says 172.21.15.153:60392) 2024-05-06T14:22:31.773 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f7fff700 1 -- 172.21.15.153:0/673537387 learned_addr learned my addr 172.21.15.153:0/673537387 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f7fff700 1 -- 172.21.15.153:0/673537387 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f81065a0 con 0x7f28f8105b10 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f7fff700 1 --2- 172.21.15.153:0/673537387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8105f20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0098a0 tx=0x7f28ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edd99c831c1234e server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f6ffd700 1 -- 172.21.15.153:0/673537387 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ec004180 con 0x7f28f8105b10 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f6ffd700 1 -- 172.21.15.153:0/673537387 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f28ec0042e0 con 0x7f28f8105b10 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.980+0000 7f28f6ffd700 1 -- 172.21.15.153:0/673537387 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ec011a50 con 0x7f28f8105b10 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 -- 172.21.15.153:0/673537387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 msgr2=0x7f28f8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 --2- 172.21.15.153:0/673537387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8105f20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0098a0 tx=0x7f28ec009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 -- 172.21.15.153:0/673537387 shutdown_connections 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 --2- 172.21.15.153:0/673537387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8105f20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 -- 172.21.15.153:0/673537387 >> 172.21.15.153:0/673537387 conn(0x7f28f8101180 msgr2=0x7f28f8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 -- 172.21.15.153:0/673537387 shutdown_connections 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.981+0000 7f28fe4aa700 1 -- 172.21.15.153:0/673537387 wait complete. 2024-05-06T14:22:31.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28fe4aa700 1 Processor -- start 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28fe4aa700 1 -- start start 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28fe4aa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28fe4aa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f8199d90 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60408/0 (socket says 172.21.15.153:60408) 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28f7fff700 1 -- 172.21.15.153:0/3872611255 learned_addr learned my addr 172.21.15.153:0/3872611255 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.982+0000 7f28f7fff700 1 -- 172.21.15.153:0/3872611255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009550 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.983+0000 7f28f7fff700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8199850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0119e0 tx=0x7f28ec011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.983+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ec003b90 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.983+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f28ec01a8a0 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.983+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f8199f90 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.983+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28ec019930 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.983+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f819a3b0 con 0x7f28f8105b10 2024-05-06T14:22:31.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.984+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f28ec019ba0 con 0x7f28f8105b10 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.984+0000 7f28f57fa700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f28e00383b0 0x7f28e003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.984+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f8062380 con 0x7f28f8105b10 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.984+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f28ec04c6e0 con 0x7f28f8105b10 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.984+0000 7f28f77fe700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f28e00383b0 0x7f28e003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.984+0000 7f28f77fe700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f28e00383b0 0x7f28e003a860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f28e8006fd0 tx=0x7f28e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:30.989+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f28ec02c3a0 con 0x7f28f8105b10 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.252+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f28f819d550 con 0x7f28f8105b10 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.252+0000 7f28f57fa700 1 -- 172.21.15.153:0/3872611255 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f28ec023390 con 0x7f28f8105b10 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f28e00383b0 msgr2=0x7f28e003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f28e00383b0 0x7f28e003a860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f28e8006fd0 tx=0x7f28e8006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:31.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 msgr2=0x7f28f8199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8199850 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0119e0 tx=0x7f28ec011ac0 comp rx=0 tx=0).stop 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 shutdown_connections 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f28e00383b0 0x7f28e003a860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 --2- 172.21.15.153:0/3872611255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28f8105b10 0x7f28f8199850 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 >> 172.21.15.153:0/3872611255 conn(0x7f28f8101180 msgr2=0x7f28f8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 shutdown_connections 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:31.255+0000 7f28fe4aa700 1 -- 172.21.15.153:0/3872611255 wait complete. 2024-05-06T14:22:31.777 INFO:teuthology.orchestra.run.smithi153.stdout:mgr is available 2024-05-06T14:22:32.186 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:31 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3872611255' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.188+0000 7fd1c8664700 1 Processor -- start 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.188+0000 7fd1c8664700 1 -- start start 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c8664700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c0104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c8664700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0105350 con 0x7fd1c0104a00 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c6400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c0104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c6400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c0104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60414/0 (socket says 172.21.15.153:60414) 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c6400700 1 -- 172.21.15.153:0/3767221733 learned_addr learned my addr 172.21.15.153:0/3767221733 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c6400700 1 -- 172.21.15.153:0/3767221733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c0105490 con 0x7fd1c0104a00 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c6400700 1 --2- 172.21.15.153:0/3767221733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c0104e10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0098a0 tx=0x7fd1bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9306564abdffb509 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.189+0000 7fd1c53fe700 1 -- 172.21.15.153:0/3767221733 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc004180 con 0x7fd1c0104a00 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c53fe700 1 -- 172.21.15.153:0/3767221733 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd1bc0042e0 con 0x7fd1c0104a00 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c53fe700 1 -- 172.21.15.153:0/3767221733 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc011a50 con 0x7fd1c0104a00 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 -- 172.21.15.153:0/3767221733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 msgr2=0x7fd1c0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 --2- 172.21.15.153:0/3767221733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c0104e10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0098a0 tx=0x7fd1bc009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 -- 172.21.15.153:0/3767221733 shutdown_connections 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 --2- 172.21.15.153:0/3767221733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c0104e10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 -- 172.21.15.153:0/3767221733 >> 172.21.15.153:0/3767221733 conn(0x7fd1c0100070 msgr2=0x7fd1c0102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 -- 172.21.15.153:0/3767221733 shutdown_connections 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.190+0000 7fd1c8664700 1 -- 172.21.15.153:0/3767221733 wait complete. 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c8664700 1 Processor -- start 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c8664700 1 -- start start 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c8664700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c8664700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c019bf30 con 0x7fd1c0104a00 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c6400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c6400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60428/0 (socket says 172.21.15.153:60428) 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c6400700 1 -- 172.21.15.153:0/2692753982 learned_addr learned my addr 172.21.15.153:0/2692753982 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c6400700 1 -- 172.21.15.153:0/2692753982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc009550 con 0x7fd1c0104a00 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.191+0000 7fd1c6400700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c019b9f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0119e0 tx=0x7fd1bc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.192+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc003b90 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.192+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd1bc01a8a0 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.192+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c019c130 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.192+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1bc019930 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.192+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c019c550 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.192+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7fd1bc019bf0 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.193+0000 7fd1b77fe700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7fd1ac038350 0x7fd1ac03a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.193+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd1bc04c870 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.193+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c0191330 con 0x7fd1c0104a00 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.193+0000 7fd1c5bff700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7fd1ac038350 0x7fd1ac03a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.193+0000 7fd1c5bff700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7fd1ac038350 0x7fd1ac03a800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0006fd0 tx=0x7fd1b0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.197+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd1bc02d440 con 0x7fd1c0104a00 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.385+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd1c002c280 con 0x7fd1c0104a00 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.386+0000 7fd1b77fe700 1 -- 172.21.15.153:0/2692753982 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fd1bc018c40 con 0x7fd1c0104a00 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7fd1ac038350 msgr2=0x7fd1ac03a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7fd1ac038350 0x7fd1ac03a800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0006fd0 tx=0x7fd1b0006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 msgr2=0x7fd1c019b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c019b9f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0119e0 tx=0x7fd1bc011ac0 comp rx=0 tx=0).stop 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 shutdown_connections 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7fd1ac038350 0x7fd1ac03a800 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 --2- 172.21.15.153:0/2692753982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1c0104a00 0x7fd1c019b9f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 >> 172.21.15.153:0/2692753982 conn(0x7fd1c0100070 msgr2=0x7fd1c018e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 shutdown_connections 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:32.388+0000 7fd1c8664700 1 -- 172.21.15.153:0/2692753982 wait complete. 2024-05-06T14:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling cephadm module... 2024-05-06T14:22:33.189 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:32 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2692753982' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T14:22:34.216 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:33 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1047194660' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-06T14:22:34.517 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f477b996700 1 Processor -- start 2024-05-06T14:22:34.517 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f477b996700 1 -- start start 2024-05-06T14:22:34.517 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f477b996700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f4774105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:34.517 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f477b996700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4774106460 con 0x7f4774105b10 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f4779732700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f4774105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f4779732700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f4774105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60438/0 (socket says 172.21.15.153:60438) 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.391+0000 7f4779732700 1 -- 172.21.15.153:0/383464808 learned_addr learned my addr 172.21.15.153:0/383464808 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f4779732700 1 -- 172.21.15.153:0/383464808 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47741065a0 con 0x7f4774105b10 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f4779732700 1 --2- 172.21.15.153:0/383464808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f4774105f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47700098a0 tx=0x7f4770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5d7e4416a70e31 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f476bfff700 1 -- 172.21.15.153:0/383464808 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4770004180 con 0x7f4774105b10 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f476bfff700 1 -- 172.21.15.153:0/383464808 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f47700042e0 con 0x7f4774105b10 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f476bfff700 1 -- 172.21.15.153:0/383464808 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4770011a50 con 0x7f4774105b10 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f477b996700 1 -- 172.21.15.153:0/383464808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 msgr2=0x7f4774105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.392+0000 7f477b996700 1 --2- 172.21.15.153:0/383464808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f4774105f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47700098a0 tx=0x7f4770009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 -- 172.21.15.153:0/383464808 shutdown_connections 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 --2- 172.21.15.153:0/383464808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f4774105f20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:34.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 -- 172.21.15.153:0/383464808 >> 172.21.15.153:0/383464808 conn(0x7f4774101180 msgr2=0x7f4774103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 -- 172.21.15.153:0/383464808 shutdown_connections 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 -- 172.21.15.153:0/383464808 wait complete. 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 Processor -- start 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 -- start start 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f477407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.393+0000 7f477b996700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477407c180 con 0x7f4774105b10 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f4779732700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f477407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f4779732700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f477407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60454/0 (socket says 172.21.15.153:60454) 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f4779732700 1 -- 172.21.15.153:0/1047194660 learned_addr learned my addr 172.21.15.153:0/1047194660 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f4779732700 1 -- 172.21.15.153:0/1047194660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770009550 con 0x7f4774105b10 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f4779732700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f477407bc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f47700119e0 tx=0x7f4770011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4770003b90 con 0x7f4774105b10 2024-05-06T14:22:34.519 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f477001a8a0 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477407a290 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4770019930 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.394+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477407a6b0 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.395+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 45073+0+0 (secure 0 0 0) 0x7f4770019ba0 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.395+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4774062380 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.395+0000 7f476a7fc700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f47600383b0 0x7f476003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.395+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f477004c890 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.395+0000 7f4778f31700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f47600383b0 0x7f476003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.396+0000 7f4778f31700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f47600383b0 0x7f476003a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4764006fd0 tx=0x7f4764006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.400+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f477002b750 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.642+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4774107980 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.952+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f477002dce0 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.952+0000 7f476a7fc700 1 -- 172.21.15.153:0/1047194660 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f477004c2b0 con 0x7f4774105b10 2024-05-06T14:22:34.520 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f47600383b0 msgr2=0x7f476003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f47600383b0 0x7f476003a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4764006fd0 tx=0x7f4764006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 msgr2=0x7f477407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f477407bc40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f47700119e0 tx=0x7f4770011ac0 comp rx=0 tx=0).stop 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 shutdown_connections 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f47600383b0 0x7f476003a860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 --2- 172.21.15.153:0/1047194660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4774105b10 0x7f477407bc40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 >> 172.21.15.153:0/1047194660 conn(0x7f4774101180 msgr2=0x7f4774101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 shutdown_connections 2024-05-06T14:22:34.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:33.955+0000 7f477b996700 1 -- 172.21.15.153:0/1047194660 wait complete. 2024-05-06T14:22:35.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:34 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1047194660' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-06T14:22:35.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:34 smithi153 ceph-mon[27177]: mgrmap e5: smithi153.dolfed(active, since 6s) 2024-05-06T14:22:35.774 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.dolfed", 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.975+0000 7f3907e3a700 1 Processor -- start 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.975+0000 7f3907e3a700 1 -- start start 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.975+0000 7f3907e3a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f3900105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.975+0000 7f3907e3a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3900105860 con 0x7f3900104f10 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3905bd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f3900105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3905bd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f3900105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60470/0 (socket says 172.21.15.153:60470) 2024-05-06T14:22:35.775 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3905bd6700 1 -- 172.21.15.153:0/4287466166 learned_addr learned my addr 172.21.15.153:0/4287466166 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3905bd6700 1 -- 172.21.15.153:0/4287466166 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39001059a0 con 0x7f3900104f10 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3905bd6700 1 --2- 172.21.15.153:0/4287466166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f3900105320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38f00098a0 tx=0x7f38f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd03c9ec13703988 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3904bd4700 1 -- 172.21.15.153:0/4287466166 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f0004180 con 0x7f3900104f10 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3904bd4700 1 -- 172.21.15.153:0/4287466166 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f38f00042e0 con 0x7f3900104f10 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.976+0000 7f3904bd4700 1 -- 172.21.15.153:0/4287466166 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f0011a50 con 0x7f3900104f10 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 -- 172.21.15.153:0/4287466166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 msgr2=0x7f3900105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 --2- 172.21.15.153:0/4287466166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f3900105320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38f00098a0 tx=0x7f38f0009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 -- 172.21.15.153:0/4287466166 shutdown_connections 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 --2- 172.21.15.153:0/4287466166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f3900105320 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 -- 172.21.15.153:0/4287466166 >> 172.21.15.153:0/4287466166 conn(0x7f39000ff7e0 msgr2=0x7f3900101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 -- 172.21.15.153:0/4287466166 shutdown_connections 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.977+0000 7f3907e3a700 1 -- 172.21.15.153:0/4287466166 wait complete. 2024-05-06T14:22:35.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3907e3a700 1 Processor -- start 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3907e3a700 1 -- start start 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3907e3a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f390018fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3907e3a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3900190110 con 0x7f3900104f10 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3905bd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f390018fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3905bd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f390018fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60472/0 (socket says 172.21.15.153:60472) 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3905bd6700 1 -- 172.21.15.153:0/3111865459 learned_addr learned my addr 172.21.15.153:0/3111865459 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3905bd6700 1 -- 172.21.15.153:0/3111865459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f0009550 con 0x7f3900104f10 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.978+0000 7f3905bd6700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f390018fbd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000c00 tx=0x7f38f0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.979+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f0003b90 con 0x7f3900104f10 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.979+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f38f00248b0 con 0x7f3900104f10 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.979+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390018e220 con 0x7f3900104f10 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.979+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f001a8e0 con 0x7f3900104f10 2024-05-06T14:22:35.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.979+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390018e640 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.980+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f38f001aba0 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.980+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3900062380 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.980+0000 7f38f6ffd700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 0x7f38ec03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.980+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f38f004c0f0 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.980+0000 7f39053d5700 1 -- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 msgr2=0x7f38ec03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.980+0000 7f39053d5700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 0x7f38ec03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:34.989+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f38f00233a0 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.181+0000 7f39053d5700 1 -- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 msgr2=0x7f38ec03a8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.181+0000 7f39053d5700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 0x7f38ec03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.271+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f39001068f0 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.272+0000 7f38f6ffd700 1 -- 172.21.15.153:0/3111865459 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f38f0018ca0 con 0x7f3900104f10 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 msgr2=0x7f38ec03a8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 0x7f38ec03a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:35.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 msgr2=0x7f390018fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f390018fbd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000c00 tx=0x7f38f0011ab0 comp rx=0 tx=0).stop 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 shutdown_connections 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f38ec0383f0 0x7f38ec03a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 --2- 172.21.15.153:0/3111865459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3900104f10 0x7f390018fbd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 >> 172.21.15.153:0/3111865459 conn(0x7f39000ff7e0 msgr2=0x7f39001043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 shutdown_connections 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:35.275+0000 7f3907e3a700 1 -- 172.21.15.153:0/3111865459 wait complete. 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-05-06T14:22:35.779 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 5... 2024-05-06T14:22:36.016 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:35 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3111865459' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T14:22:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: Active manager daemon smithi153.dolfed restarted 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: Activating manager daemon smithi153.dolfed 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: osdmap e2: 0 total, 0 up, 0 in 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: mgrmap e6: smithi153.dolfed(active, starting, since 0.00319026s) 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr metadata", "who": "smithi153.dolfed", "id": "smithi153.dolfed"}]: dispatch 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: Manager daemon smithi153.dolfed is now available 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:40.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:22:40.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:22:40.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:22:40.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/mirror_snapshot_schedule"}]: dispatch 2024-05-06T14:22:40.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:40 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/trash_purge_schedule"}]: dispatch 2024-05-06T14:22:41.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:41 smithi153 ceph-mon[27177]: Found migration_current of "None". Setting to last migration. 2024-05-06T14:22:41.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:41 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:41.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:41 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:41.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:41 smithi153 ceph-mon[27177]: mgrmap e7: smithi153.dolfed(active, since 1.00522s) 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f85349b1700 1 Processor -- start 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f85349b1700 1 -- start start 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f85349b1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f8530104e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f85349b1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8530105350 con 0x7f8530104a00 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f852e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f8530104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f852e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f8530104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60482/0 (socket says 172.21.15.153:60482) 2024-05-06T14:22:42.133 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.248+0000 7f852e59c700 1 -- 172.21.15.153:0/2054956243 learned_addr learned my addr 172.21.15.153:0/2054956243 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.249+0000 7f852e59c700 1 -- 172.21.15.153:0/2054956243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530105490 con 0x7f8530104a00 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.249+0000 7f852e59c700 1 --2- 172.21.15.153:0/2054956243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f8530104e10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f852000bb40 tx=0x7f852000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b48d3164712fca6 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.249+0000 7f852d59a700 1 -- 172.21.15.153:0/2054956243 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f852000dc00 con 0x7f8530104a00 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.249+0000 7f852d59a700 1 -- 172.21.15.153:0/2054956243 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f85200044a0 con 0x7f8530104a00 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.249+0000 7f852d59a700 1 -- 172.21.15.153:0/2054956243 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8520010470 con 0x7f8530104a00 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 -- 172.21.15.153:0/2054956243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 msgr2=0x7f8530104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 --2- 172.21.15.153:0/2054956243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f8530104e10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f852000bb40 tx=0x7f852000be50 comp rx=0 tx=0).stop 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 -- 172.21.15.153:0/2054956243 shutdown_connections 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 --2- 172.21.15.153:0/2054956243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f8530104e10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 -- 172.21.15.153:0/2054956243 >> 172.21.15.153:0/2054956243 conn(0x7f8530100070 msgr2=0x7f8530102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 -- 172.21.15.153:0/2054956243 shutdown_connections 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.250+0000 7f85349b1700 1 -- 172.21.15.153:0/2054956243 wait complete. 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f85349b1700 1 Processor -- start 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f85349b1700 1 -- start start 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f85349b1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f853019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f85349b1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853019bf30 con 0x7f8530104a00 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f852e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f853019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f852e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f853019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60488/0 (socket says 172.21.15.153:60488) 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f852e59c700 1 -- 172.21.15.153:0/4268550228 learned_addr learned my addr 172.21.15.153:0/4268550228 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f852e59c700 1 -- 172.21.15.153:0/4268550228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852000b7f0 con 0x7f8530104a00 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.251+0000 7f852e59c700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f853019b9f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f852000dfa0 tx=0x7f8520003fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.252+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85200104e0 con 0x7f8530104a00 2024-05-06T14:22:42.135 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.252+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8520010ba0 con 0x7f8530104a00 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.252+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8520021c30 con 0x7f8530104a00 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.252+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853019c130 con 0x7f8530104a00 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.252+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853019c4f0 con 0x7f8530104a00 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.252+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f85200193f0 con 0x7f8530104a00 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.253+0000 7f851f7fe700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.253+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 --> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8530073760 con 0x7f85180383a0 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.253+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f852004cbb0 con 0x7f8530104a00 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.253+0000 7f852dd9b700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 msgr2=0x7f851803a850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.253+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.454+0000 7f852dd9b700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 msgr2=0x7f851803a850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.454+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.855+0000 7f852dd9b700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 msgr2=0x7f851803a850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:36.855+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-06T14:22:42.136 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:37.656+0000 7f852dd9b700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 msgr2=0x7f851803a850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:37.656+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:39.257+0000 7f852dd9b700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 msgr2=0x7f851803a850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4102256583 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:39.257+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:40.570+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 6) v1 ==== 44851+0+0 (secure 0 0 0) 0x7f85200203c0 con 0x7f8530104a00 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:40.570+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 msgr2=0x7f851803a850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:40.570+0000 7f851f7fe700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.571+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7f852000f3d0 con 0x7f8530104a00 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.572+0000 7f851f7fe700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f851803bb10 0x7f851803def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.572+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f851803e430 con 0x7f851803bb10 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.572+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f851803bb10 0x7f851803def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.573+0000 7f852dd9b700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f851803bb10 0x7f851803def0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8524003a10 tx=0x7f85240092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.573+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f851803e430 con 0x7f851803bb10 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.579+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f853019cd30 con 0x7f851803bb10 2024-05-06T14:22:42.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f851f7fe700 1 -- 172.21.15.153:0/4268550228 <== mgr.14120 v2:172.21.15.153:6800/585874296 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f853019cd30 con 0x7f851803bb10 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f851803bb10 msgr2=0x7f851803def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f851803bb10 0x7f851803def0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8524003a10 tx=0x7f85240092b0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 msgr2=0x7f853019b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f853019b9f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f852000dfa0 tx=0x7f8520003fa0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 shutdown_connections 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f851803bb10 0x7f851803def0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8530104a00 0x7f853019b9f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 --2- 172.21.15.153:0/4268550228 >> [v2:172.21.15.153:6800/4102256583,v1:172.21.15.153:6801/4102256583] conn(0x7f85180383a0 0x7f851803a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 >> 172.21.15.153:0/4268550228 conn(0x7f8530100070 msgr2=0x7f853018e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 shutdown_connections 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:41.580+0000 7f85349b1700 1 -- 172.21.15.153:0/4268550228 wait complete. 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 5 is available 2024-05-06T14:22:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:Setting orchestrator backend to cephadm... 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: [06/May/2024:14:22:41] ENGINE Bus STARTING 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: [06/May/2024:14:22:41] ENGINE Serving on http://172.21.15.153:8765 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: [06/May/2024:14:22:41] ENGINE Serving on https://172.21.15.153:7150 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: [06/May/2024:14:22:41] ENGINE Bus STARTED 2024-05-06T14:22:42.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:42 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:22:43.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.568+0000 7f9333aaf700 1 Processor -- start 2024-05-06T14:22:43.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f9333aaf700 1 -- start start 2024-05-06T14:22:43.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f9333aaf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:43.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f9333aaf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0744b0 con 0x7f932c106670 2024-05-06T14:22:43.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f933184b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:43.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f933184b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50192/0 (socket says 172.21.15.153:50192) 2024-05-06T14:22:43.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f933184b700 1 -- 172.21.15.153:0/2389671537 learned_addr learned my addr 172.21.15.153:0/2389671537 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:43.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f933184b700 1 -- 172.21.15.153:0/2389671537 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c0745f0 con 0x7f932c106670 2024-05-06T14:22:43.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.569+0000 7f933184b700 1 --2- 172.21.15.153:0/2389671537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c108a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f93280098a0 tx=0x7f9328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e0884086a326c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:43.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9330849700 1 -- 172.21.15.153:0/2389671537 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9328004180 con 0x7f932c106670 2024-05-06T14:22:43.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9330849700 1 -- 172.21.15.153:0/2389671537 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93280042e0 con 0x7f932c106670 2024-05-06T14:22:43.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9330849700 1 -- 172.21.15.153:0/2389671537 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9328011a50 con 0x7f932c106670 2024-05-06T14:22:43.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 -- 172.21.15.153:0/2389671537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 msgr2=0x7f932c108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:43.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 --2- 172.21.15.153:0/2389671537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c108a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f93280098a0 tx=0x7f9328009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:43.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 -- 172.21.15.153:0/2389671537 shutdown_connections 2024-05-06T14:22:43.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 --2- 172.21.15.153:0/2389671537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c108a50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:43.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 -- 172.21.15.153:0/2389671537 >> 172.21.15.153:0/2389671537 conn(0x7f932c100070 msgr2=0x7f932c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 -- 172.21.15.153:0/2389671537 shutdown_connections 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.570+0000 7f9333aaf700 1 -- 172.21.15.153:0/2389671537 wait complete. 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f9333aaf700 1 Processor -- start 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f9333aaf700 1 -- start start 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f9333aaf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c105fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f9333aaf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c106510 con 0x7f932c106670 2024-05-06T14:22:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f933184b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c105fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:43.301 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f933184b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c105fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50198/0 (socket says 172.21.15.153:50198) 2024-05-06T14:22:43.301 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f933184b700 1 -- 172.21.15.153:0/3506333954 learned_addr learned my addr 172.21.15.153:0/3506333954 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:43.301 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.571+0000 7f933184b700 1 -- 172.21.15.153:0/3506333954 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328009550 con 0x7f932c106670 2024-05-06T14:22:43.301 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.572+0000 7f933184b700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c105fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f9328011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:43.301 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.572+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9328003b90 con 0x7f932c106670 2024-05-06T14:22:43.301 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.572+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f93280248b0 con 0x7f932c106670 2024-05-06T14:22:43.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.572+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c104620 con 0x7f932c106670 2024-05-06T14:22:43.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.572+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f932801a8e0 con 0x7f932c106670 2024-05-06T14:22:43.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.572+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c104a40 con 0x7f932c106670 2024-05-06T14:22:43.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.573+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 7) v1 ==== 44978+0+0 (secure 0 0 0) 0x7f932801ab50 con 0x7f932c106670 2024-05-06T14:22:43.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.573+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c109b80 con 0x7f932c106670 2024-05-06T14:22:43.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.573+0000 7f9322ffd700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9318038320 0x7f931803a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:43.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.573+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f932804c3d0 con 0x7f932c106670 2024-05-06T14:22:43.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.573+0000 7f933104a700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9318038320 0x7f931803a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:43.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.574+0000 7f933104a700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9318038320 0x7f931803a7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f931c006fd0 tx=0x7f931c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:43.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.578+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f93280189a0 con 0x7f932c106670 2024-05-06T14:22:43.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.625+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f932802dc90 con 0x7f932c106670 2024-05-06T14:22:43.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.782+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f932c102370 con 0x7f9318038320 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.789+0000 7f9322ffd700 1 -- 172.21.15.153:0/3506333954 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f932c102370 con 0x7f9318038320 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9318038320 msgr2=0x7f931803a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9318038320 0x7f931803a7d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f931c006fd0 tx=0x7f931c006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 msgr2=0x7f932c105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c105fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9328000c00 tx=0x7f9328011ab0 comp rx=0 tx=0).stop 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 shutdown_connections 2024-05-06T14:22:43.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9318038320 0x7f931803a7d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:43.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.791+0000 7f9333aaf700 1 --2- 172.21.15.153:0/3506333954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c106670 0x7f932c105fd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:43.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.792+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 >> 172.21.15.153:0/3506333954 conn(0x7f932c100070 msgr2=0x7f932c101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:43.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.792+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 shutdown_connections 2024-05-06T14:22:43.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:42.792+0000 7f9333aaf700 1 -- 172.21.15.153:0/3506333954 wait complete. 2024-05-06T14:22:43.646 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:43 smithi153 ceph-mon[27177]: mgrmap e8: smithi153.dolfed(active, since 2s) 2024-05-06T14:22:43.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:43 smithi153 ceph-mon[27177]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:43.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:43 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:43.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:43 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:22:44.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f05964da700 1 Processor -- start 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f05964da700 1 -- start start 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f05964da700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f05964da700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0590106460 con 0x7f0590105b10 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f058ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f058ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50202/0 (socket says 172.21.15.153:50202) 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f058ffff700 1 -- 172.21.15.153:0/311393951 learned_addr learned my addr 172.21.15.153:0/311393951 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.658+0000 7f058ffff700 1 -- 172.21.15.153:0/311393951 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05901065a0 con 0x7f0590105b10 2024-05-06T14:22:44.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f058ffff700 1 --2- 172.21.15.153:0/311393951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590105f20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f05800098a0 tx=0x7f0580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f4215587a10a5e server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f058effd700 1 -- 172.21.15.153:0/311393951 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0580004180 con 0x7f0590105b10 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f058effd700 1 -- 172.21.15.153:0/311393951 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f05800042e0 con 0x7f0590105b10 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f058effd700 1 -- 172.21.15.153:0/311393951 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0580011a50 con 0x7f0590105b10 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 -- 172.21.15.153:0/311393951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 msgr2=0x7f0590105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 --2- 172.21.15.153:0/311393951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590105f20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f05800098a0 tx=0x7f0580009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 -- 172.21.15.153:0/311393951 shutdown_connections 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 --2- 172.21.15.153:0/311393951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590105f20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 -- 172.21.15.153:0/311393951 >> 172.21.15.153:0/311393951 conn(0x7f0590101180 msgr2=0x7f0590103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 -- 172.21.15.153:0/311393951 shutdown_connections 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.659+0000 7f05964da700 1 -- 172.21.15.153:0/311393951 wait complete. 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f05964da700 1 Processor -- start 2024-05-06T14:22:44.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f05964da700 1 -- start start 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f05964da700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f05964da700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0590199d50 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f058ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f058ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50204/0 (socket says 172.21.15.153:50204) 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.660+0000 7f058ffff700 1 -- 172.21.15.153:0/2914767118 learned_addr learned my addr 172.21.15.153:0/2914767118 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f058ffff700 1 -- 172.21.15.153:0/2914767118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580009550 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f058ffff700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590199810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05800119e0 tx=0x7f0580011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0580003b90 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0590199f50 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f058001a8a0 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0580019930 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.661+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059019a370 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.662+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f0580019bf0 con 0x7f0590105b10 2024-05-06T14:22:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.662+0000 7f058d7fa700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f05780383b0 0x7f057803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.662+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0590062380 con 0x7f0590105b10 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.662+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f058004c710 con 0x7f0590105b10 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.662+0000 7f058f7fe700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f05780383b0 0x7f057803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.662+0000 7f058f7fe700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f05780383b0 0x7f057803a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0584006fd0 tx=0x7f0584006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.667+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f058002d440 con 0x7f0590105b10 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.857+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0590062380 con 0x7f05780383b0 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.858+0000 7f058d7fa700 1 -- 172.21.15.153:0/2914767118 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0590062380 con 0x7f05780383b0 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f05780383b0 msgr2=0x7f057803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f05780383b0 0x7f057803a860 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0584006fd0 tx=0x7f0584006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 msgr2=0x7f0590199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590199810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05800119e0 tx=0x7f0580011ac0 comp rx=0 tx=0).stop 2024-05-06T14:22:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 shutdown_connections 2024-05-06T14:22:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f05780383b0 0x7f057803a860 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 --2- 172.21.15.153:0/2914767118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0590105b10 0x7f0590199810 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 >> 172.21.15.153:0/2914767118 conn(0x7f0590101180 msgr2=0x7f0590101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 shutdown_connections 2024-05-06T14:22:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:43.860+0000 7f05964da700 1 -- 172.21.15.153:0/2914767118 wait complete. 2024-05-06T14:22:44.440 INFO:teuthology.orchestra.run.smithi153.stdout:Generating ssh key... 2024-05-06T14:22:45.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:44 smithi153 ceph-mon[27177]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:45.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf8bf48700 1 Processor -- start 2024-05-06T14:22:45.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf8bf48700 1 -- start start 2024-05-06T14:22:45.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf8bf48700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf841051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:45.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf8bf48700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf841057a0 con 0x7fcf84104da0 2024-05-06T14:22:45.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf89ce4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf841051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf89ce4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf841051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50214/0 (socket says 172.21.15.153:50214) 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf89ce4700 1 -- 172.21.15.153:0/3617426911 learned_addr learned my addr 172.21.15.153:0/3617426911 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.921+0000 7fcf89ce4700 1 -- 172.21.15.153:0/3617426911 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf841058e0 con 0x7fcf84104da0 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf89ce4700 1 --2- 172.21.15.153:0/3617426911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf841051b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcf800098a0 tx=0x7fcf80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d293832e5ebfe1 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf88ce2700 1 -- 172.21.15.153:0/3617426911 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf80004180 con 0x7fcf84104da0 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf88ce2700 1 -- 172.21.15.153:0/3617426911 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcf800042e0 con 0x7fcf84104da0 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf88ce2700 1 -- 172.21.15.153:0/3617426911 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf80011a50 con 0x7fcf84104da0 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 -- 172.21.15.153:0/3617426911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 msgr2=0x7fcf841051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 --2- 172.21.15.153:0/3617426911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf841051b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcf800098a0 tx=0x7fcf80009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 -- 172.21.15.153:0/3617426911 shutdown_connections 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 --2- 172.21.15.153:0/3617426911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf841051b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 -- 172.21.15.153:0/3617426911 >> 172.21.15.153:0/3617426911 conn(0x7fcf840ff3d0 msgr2=0x7fcf841017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:45.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 -- 172.21.15.153:0/3617426911 shutdown_connections 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.922+0000 7fcf8bf48700 1 -- 172.21.15.153:0/3617426911 wait complete. 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf8bf48700 1 Processor -- start 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf8bf48700 1 -- start start 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf8bf48700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf8419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf8bf48700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8419bef0 con 0x7fcf84104da0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf89ce4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf8419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf89ce4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf8419b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50230/0 (socket says 172.21.15.153:50230) 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.923+0000 7fcf89ce4700 1 -- 172.21.15.153:0/2821582064 learned_addr learned my addr 172.21.15.153:0/2821582064 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf89ce4700 1 -- 172.21.15.153:0/2821582064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf80009550 con 0x7fcf84104da0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf89ce4700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf8419b9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf80006af0 tx=0x7fcf80011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf80003b90 con 0x7fcf84104da0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcf8001a8a0 con 0x7fcf84104da0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8419c0f0 con 0x7fcf84104da0 2024-05-06T14:22:45.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf80019880 con 0x7fcf84104da0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.924+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8419c510 con 0x7fcf84104da0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.925+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fcf8001a430 con 0x7fcf84104da0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.925+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf84195340 con 0x7fcf84104da0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.925+0000 7fcf7affd700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fcf700383b0 0x7fcf7003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.925+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcf8004c6e0 con 0x7fcf84104da0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.925+0000 7fcf894e3700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fcf700383b0 0x7fcf7003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.925+0000 7fcf894e3700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fcf700383b0 0x7fcf7003a860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcf74006fd0 tx=0x7fcf74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:44.930+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcf80019b40 con 0x7fcf84104da0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.122+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf84061190 con 0x7fcf700383b0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.351+0000 7fcf7affd700 1 -- 172.21.15.153:0/2821582064 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcf84061190 con 0x7fcf700383b0 2024-05-06T14:22:45.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fcf700383b0 msgr2=0x7fcf7003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fcf700383b0 0x7fcf7003a860 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcf74006fd0 tx=0x7fcf74006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 msgr2=0x7fcf8419b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf8419b9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf80006af0 tx=0x7fcf80011a90 comp rx=0 tx=0).stop 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 shutdown_connections 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fcf700383b0 0x7fcf7003a860 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 --2- 172.21.15.153:0/2821582064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf84104da0 0x7fcf8419b9b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 >> 172.21.15.153:0/2821582064 conn(0x7fcf840ff3d0 msgr2=0x7fcf84192480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 shutdown_connections 2024-05-06T14:22:45.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:45.354+0000 7fcf8bf48700 1 -- 172.21.15.153:0/2821582064 wait complete. 2024-05-06T14:22:46.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:46 smithi153 ceph-mon[27177]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:46.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:46 smithi153 ceph-mon[27177]: Generating ssh key... 2024-05-06T14:22:46.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:46 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:46.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:46 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:47.137 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS66Y9i5ffT8perfG2ahf3etinZhNNBXz4H0GeBuHxg+d2GoqSQfUcNATCCs8V/WjeEI4XUBxNo8ghN/L8RLlK+bN6/20Ab2fcfXGTHGDVO4faVnI+tMAhdTfwyuquNEc8BWKj4uZZTn77YhXlp8/ibSNZFAZbs8zSOHQpFKZUR3JyOiYIuLvB14KWXCAk75ZZ1IckF7TYWv7AUZTO+EIoisifBYQZcPDFjtLLNQObo9um7EtnXEhMCKao7QXk+G/8M0KpksUDEvxXbi2pPXPnCUHzpUWvP+stnDe5kNmqLIFR16HAzt1+bkKcEbckLNcULYZ+SmC92x62v4uDhfM2+Pr4lEsnF/M20L2V7nNrxjqZqYRTXxhNizSBHqOzdGQl4DPETLfw3oxifrjfVqr/yyyRZN8F9cLXWnDx1uR/pV1HDsRKeU23szG5SCV1LMTOw3FHP0ajho2HvkUoVsqlHBfJfc7btjxXBe7iL1hNtPwrpkpK846oP542Lgkl7L8= ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97c0731700 1 Processor -- start 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97c0731700 1 -- start start 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97c0731700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97c0731700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8106460 con 0x7f97b8105b10 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97be4cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97be4cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50238/0 (socket says 172.21.15.153:50238) 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.375+0000 7f97be4cd700 1 -- 172.21.15.153:0/885670188 learned_addr learned my addr 172.21.15.153:0/885670188 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.376+0000 7f97be4cd700 1 -- 172.21.15.153:0/885670188 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b81065a0 con 0x7f97b8105b10 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.376+0000 7f97be4cd700 1 --2- 172.21.15.153:0/885670188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8105f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97b40098a0 tx=0x7f97b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1860505c551f6f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.376+0000 7f97bd4cb700 1 -- 172.21.15.153:0/885670188 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b4004180 con 0x7f97b8105b10 2024-05-06T14:22:47.138 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.376+0000 7f97bd4cb700 1 -- 172.21.15.153:0/885670188 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f97b40042e0 con 0x7f97b8105b10 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.376+0000 7f97bd4cb700 1 -- 172.21.15.153:0/885670188 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b4011a50 con 0x7f97b8105b10 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 -- 172.21.15.153:0/885670188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 msgr2=0x7f97b8105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 --2- 172.21.15.153:0/885670188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8105f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97b40098a0 tx=0x7f97b4009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 -- 172.21.15.153:0/885670188 shutdown_connections 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 --2- 172.21.15.153:0/885670188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8105f20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 -- 172.21.15.153:0/885670188 >> 172.21.15.153:0/885670188 conn(0x7f97b8101180 msgr2=0x7f97b8103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 -- 172.21.15.153:0/885670188 shutdown_connections 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 -- 172.21.15.153:0/885670188 wait complete. 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 Processor -- start 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 -- start start 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.377+0000 7f97c0731700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97c0731700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8199d50 con 0x7f97b8105b10 2024-05-06T14:22:47.139 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97be4cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97be4cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50244/0 (socket says 172.21.15.153:50244) 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97be4cd700 1 -- 172.21.15.153:0/394709946 learned_addr learned my addr 172.21.15.153:0/394709946 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97be4cd700 1 -- 172.21.15.153:0/394709946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4009550 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97be4cd700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8199810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f97b40119e0 tx=0x7f97b4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b4003b90 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f97b401a8a0 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b4019930 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b8199f50 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.378+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b819a370 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.379+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f97b4019ba0 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.379+0000 7f97af7fe700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f97a4038400 0x7f97a403a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.379+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b8062380 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.379+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f97b404c8b0 con 0x7f97b8105b10 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.379+0000 7f97bdccc700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f97a4038400 0x7f97a403a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:47.140 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.380+0000 7f97bdccc700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f97a4038400 0x7f97a403a8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f97a8006fd0 tx=0x7f97a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.384+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f97b401a430 con 0x7f97b8105b10 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.576+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f97b8102610 con 0x7f97a4038400 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.577+0000 7f97af7fe700 1 -- 172.21.15.153:0/394709946 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f97b8102610 con 0x7f97a4038400 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f97a4038400 msgr2=0x7f97a403a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f97a4038400 0x7f97a403a8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f97a8006fd0 tx=0x7f97a8006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 msgr2=0x7f97b8199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8199810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f97b40119e0 tx=0x7f97b4011ac0 comp rx=0 tx=0).stop 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 shutdown_connections 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f97a4038400 0x7f97a403a8b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 --2- 172.21.15.153:0/394709946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97b8105b10 0x7f97b8199810 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 >> 172.21.15.153:0/394709946 conn(0x7f97b8101180 msgr2=0x7f97b8101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 shutdown_connections 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:46.579+0000 7f97c0731700 1 -- 172.21.15.153:0/394709946 wait complete. 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-06T14:22:47.141 INFO:teuthology.orchestra.run.smithi153.stdout:Adding key to root@localhost authorized_keys... 2024-05-06T14:22:47.142 INFO:teuthology.orchestra.run.smithi153.stdout:Adding host smithi153... 2024-05-06T14:22:47.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:47 smithi153 ceph-mon[27177]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:48.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:48 smithi153 ceph-mon[27177]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:50.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:49 smithi153 ceph-mon[27177]: Deploying cephadm binary to smithi153 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Added host 'smithi153' with addr '172.21.15.153' 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.562+0000 7f2e1a979700 1 Processor -- start 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.562+0000 7f2e1a979700 1 -- start start 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.562+0000 7f2e1a979700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e14104850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e1a979700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140744d0 con 0x7f2e14104440 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e14104850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e14104850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50258/0 (socket says 172.21.15.153:50258) 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e13fff700 1 -- 172.21.15.153:0/1089467537 learned_addr learned my addr 172.21.15.153:0/1089467537 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e13fff700 1 -- 172.21.15.153:0/1089467537 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14074610 con 0x7f2e14104440 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e13fff700 1 --2- 172.21.15.153:0/1089467537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e14104850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098a0 tx=0x7f2e04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef21734f3dd94f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e12ffd700 1 -- 172.21.15.153:0/1089467537 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e04004180 con 0x7f2e14104440 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e12ffd700 1 -- 172.21.15.153:0/1089467537 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e040042e0 con 0x7f2e14104440 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.563+0000 7f2e12ffd700 1 -- 172.21.15.153:0/1089467537 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e04011a50 con 0x7f2e14104440 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 -- 172.21.15.153:0/1089467537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 msgr2=0x7f2e14104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 --2- 172.21.15.153:0/1089467537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e14104850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098a0 tx=0x7f2e04009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 -- 172.21.15.153:0/1089467537 shutdown_connections 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 --2- 172.21.15.153:0/1089467537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e14104850 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 -- 172.21.15.153:0/1089467537 >> 172.21.15.153:0/1089467537 conn(0x7f2e14100070 msgr2=0x7f2e14102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:50.715 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 -- 172.21.15.153:0/1089467537 shutdown_connections 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 -- 172.21.15.153:0/1089467537 wait complete. 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.564+0000 7f2e1a979700 1 Processor -- start 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e1a979700 1 -- start start 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e1a979700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e1419b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e1a979700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1419bf30 con 0x7f2e14104440 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e1419b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e1419b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50270/0 (socket says 172.21.15.153:50270) 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e13fff700 1 -- 172.21.15.153:0/1631789217 learned_addr learned my addr 172.21.15.153:0/1631789217 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e13fff700 1 -- 172.21.15.153:0/1631789217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e04009550 con 0x7f2e14104440 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e13fff700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e1419b9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2e04006af0 tx=0x7f2e04011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e04003b90 con 0x7f2e14104440 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.565+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e0401a8a0 con 0x7f2e14104440 2024-05-06T14:22:50.716 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.566+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1419c190 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.566+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e04019880 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.566+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1419c610 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.566+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f2e0401a430 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.567+0000 7f2e117fa700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f2e00038400 0x7f2e0003a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.567+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e141951a0 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.567+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2e0404c7e0 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.567+0000 7f2e137fe700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f2e00038400 0x7f2e0003a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.567+0000 7f2e137fe700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f2e00038400 0x7f2e0003a8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc006fd0 tx=0x7f2dfc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.571+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2e040189a0 con 0x7f2e14104440 2024-05-06T14:22:50.717 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:47.764+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e14061190 con 0x7f2e00038400 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.097+0000 7f2e117fa700 1 -- 172.21.15.153:0/1631789217 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2e14061190 con 0x7f2e00038400 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.099+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f2e00038400 msgr2=0x7f2e0003a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.099+0000 7f2e1a979700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f2e00038400 0x7f2e0003a8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc006fd0 tx=0x7f2dfc006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.099+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 msgr2=0x7f2e1419b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.099+0000 7f2e1a979700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e1419b9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2e04006af0 tx=0x7f2e04011a90 comp rx=0 tx=0).stop 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.100+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 shutdown_connections 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.100+0000 7f2e1a979700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f2e00038400 0x7f2e0003a8b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.100+0000 7f2e1a979700 1 --2- 172.21.15.153:0/1631789217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e14104440 0x7f2e1419b9f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.100+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 >> 172.21.15.153:0/1631789217 conn(0x7f2e14100070 msgr2=0x7f2e14190000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.100+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 shutdown_connections 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:50.100+0000 7f2e1a979700 1 -- 172.21.15.153:0/1631789217 wait complete. 2024-05-06T14:22:50.718 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mon service with default placement... 2024-05-06T14:22:51.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:51 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:51.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:51 smithi153 ceph-mon[27177]: Added host smithi153 2024-05-06T14:22:51.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:51 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:22:52.163 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-06T14:22:52.163 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f276bd00700 1 Processor -- start 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f276bd00700 1 -- start start 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f276bd00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f27641075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f276bd00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2764107b00 con 0x7f27641071b0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f2769a9c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f27641075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f2769a9c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f27641075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57548/0 (socket says 172.21.15.153:57548) 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.222+0000 7f2769a9c700 1 -- 172.21.15.153:0/1232675946 learned_addr learned my addr 172.21.15.153:0/1232675946 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.223+0000 7f2769a9c700 1 -- 172.21.15.153:0/1232675946 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764107c40 con 0x7f27641071b0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.223+0000 7f2769a9c700 1 --2- 172.21.15.153:0/1232675946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f27641075c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27540098a0 tx=0x7f2754009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb01735ccc77c37d server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.223+0000 7f2768a9a700 1 -- 172.21.15.153:0/1232675946 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2754004180 con 0x7f27641071b0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.223+0000 7f2768a9a700 1 -- 172.21.15.153:0/1232675946 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f27540042e0 con 0x7f27641071b0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.223+0000 7f2768a9a700 1 -- 172.21.15.153:0/1232675946 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2754011a50 con 0x7f27641071b0 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- 172.21.15.153:0/1232675946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 msgr2=0x7f27641075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 --2- 172.21.15.153:0/1232675946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f27641075c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27540098a0 tx=0x7f2754009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:52.164 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- 172.21.15.153:0/1232675946 shutdown_connections 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 --2- 172.21.15.153:0/1232675946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f27641075c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- 172.21.15.153:0/1232675946 >> 172.21.15.153:0/1232675946 conn(0x7f27640ff7e0 msgr2=0x7f2764101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- 172.21.15.153:0/1232675946 shutdown_connections 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- 172.21.15.153:0/1232675946 wait complete. 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 Processor -- start 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- start start 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f2764199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.224+0000 7f276bd00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2764199d70 con 0x7f27641071b0 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f2769a9c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f2764199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f2769a9c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f2764199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57552/0 (socket says 172.21.15.153:57552) 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f2769a9c700 1 -- 172.21.15.153:0/3829702431 learned_addr learned my addr 172.21.15.153:0/3829702431 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f2769a9c700 1 -- 172.21.15.153:0/3829702431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2754009550 con 0x7f27641071b0 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f2769a9c700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f2764199830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2754000c00 tx=0x7f2754011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2754003b90 con 0x7f27641071b0 2024-05-06T14:22:52.165 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2764199f70 con 0x7f27641071b0 2024-05-06T14:22:52.166 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f27540248b0 con 0x7f27641071b0 2024-05-06T14:22:52.166 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f275401a8e0 con 0x7f27641071b0 2024-05-06T14:22:52.166 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.225+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276419a390 con 0x7f27641071b0 2024-05-06T14:22:52.166 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.226+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f275401aba0 con 0x7f27641071b0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.226+0000 7f275affd700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f27500383f0 0x7f275003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.226+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2764062380 con 0x7f27641071b0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.226+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f275404c620 con 0x7f27641071b0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.226+0000 7f276929b700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f27500383f0 0x7f275003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.227+0000 7f276929b700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f27500383f0 0x7f275003a8a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2760006fd0 tx=0x7f2760006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.231+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2754024db0 con 0x7f27641071b0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.441+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f2764100cd0 con 0x7f27500383f0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.447+0000 7f275affd700 1 -- 172.21.15.153:0/3829702431 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2764100cd0 con 0x7f27500383f0 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f27500383f0 msgr2=0x7f275003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:52.167 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f27500383f0 0x7f275003a8a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2760006fd0 tx=0x7f2760006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 msgr2=0x7f2764199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f2764199830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2754000c00 tx=0x7f2754011ab0 comp rx=0 tx=0).stop 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 shutdown_connections 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f27500383f0 0x7f275003a8a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 --2- 172.21.15.153:0/3829702431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27641071b0 0x7f2764199830 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 >> 172.21.15.153:0/3829702431 conn(0x7f27640ff7e0 msgr2=0x7f27641005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 shutdown_connections 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:51.449+0000 7f276bd00700 1 -- 172.21.15.153:0/3829702431 wait complete. 2024-05-06T14:22:52.168 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mgr service with default placement... 2024-05-06T14:22:52.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:52 smithi153 ceph-mon[27177]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:52.701 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:52 smithi153 ceph-mon[27177]: Saving service mon spec with placement count:5 2024-05-06T14:22:52.701 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:52 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:53.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:53 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:53.975 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:53 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:53.975 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:53 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:53.982 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-06T14:22:53.982 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.160+0000 7f8600b15700 1 Processor -- start 2024-05-06T14:22:53.982 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.160+0000 7f8600b15700 1 -- start start 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.160+0000 7f8600b15700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc108a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.160+0000 7f8600b15700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc079d10 con 0x7f85fc1066a0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85fa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc108a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85fa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc108a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57564/0 (socket says 172.21.15.153:57564) 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85fa59c700 1 -- 172.21.15.153:0/21099447 learned_addr learned my addr 172.21.15.153:0/21099447 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85fa59c700 1 -- 172.21.15.153:0/21099447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc079e50 con 0x7f85fc1066a0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85fa59c700 1 --2- 172.21.15.153:0/21099447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc108a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0098a0 tx=0x7f85ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37dace1a1cb79e1b server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85f959a700 1 -- 172.21.15.153:0/21099447 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85ec004180 con 0x7f85fc1066a0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85f959a700 1 -- 172.21.15.153:0/21099447 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f85ec0042e0 con 0x7f85fc1066a0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.161+0000 7f85f959a700 1 -- 172.21.15.153:0/21099447 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85ec011af0 con 0x7f85fc1066a0 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 -- 172.21.15.153:0/21099447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 msgr2=0x7f85fc108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 --2- 172.21.15.153:0/21099447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc108a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0098a0 tx=0x7f85ec009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:53.983 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 -- 172.21.15.153:0/21099447 shutdown_connections 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 --2- 172.21.15.153:0/21099447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc108a80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 -- 172.21.15.153:0/21099447 >> 172.21.15.153:0/21099447 conn(0x7f85fc100060 msgr2=0x7f85fc1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 -- 172.21.15.153:0/21099447 shutdown_connections 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.162+0000 7f8600b15700 1 -- 172.21.15.153:0/21099447 wait complete. 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f8600b15700 1 Processor -- start 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f8600b15700 1 -- start start 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f8600b15700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f8600b15700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc199e70 con 0x7f85fc1066a0 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f85fa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f85fa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57570/0 (socket says 172.21.15.153:57570) 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.163+0000 7f85fa59c700 1 -- 172.21.15.153:0/4112302544 learned_addr learned my addr 172.21.15.153:0/4112302544 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f85fa59c700 1 -- 172.21.15.153:0/4112302544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec009550 con 0x7f85fc1066a0 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f85fa59c700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc199930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f85ec011670 tx=0x7f85ec011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85ec0039f0 con 0x7f85fc1066a0 2024-05-06T14:22:53.984 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f85ec003b50 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85ec0247c0 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc19a070 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.164+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc19a490 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.165+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f85ec01ac00 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.165+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc193950 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.165+0000 7f85eb7fe700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f85e4038440 0x7f85e403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.166+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f85ec04c3f0 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.166+0000 7f85f9d9b700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f85e4038440 0x7f85e403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.166+0000 7f85f9d9b700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f85e4038440 0x7f85e403a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f85f0006fd0 tx=0x7f85f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.171+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f85ec010380 con 0x7f85fc1066a0 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.406+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f85fc061190 con 0x7f85e4038440 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.411+0000 7f85eb7fe700 1 -- 172.21.15.153:0/4112302544 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f85fc061190 con 0x7f85e4038440 2024-05-06T14:22:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f85e4038440 msgr2=0x7f85e403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f85e4038440 0x7f85e403a8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f85f0006fd0 tx=0x7f85f0006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 msgr2=0x7f85fc199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc199930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f85ec011670 tx=0x7f85ec011b50 comp rx=0 tx=0).stop 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 shutdown_connections 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f85e4038440 0x7f85e403a8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 --2- 172.21.15.153:0/4112302544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85fc1066a0 0x7f85fc199930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 >> 172.21.15.153:0/4112302544 conn(0x7f85fc100060 msgr2=0x7f85fc1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.413+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 shutdown_connections 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:53.414+0000 7f8600b15700 1 -- 172.21.15.153:0/4112302544 wait complete. 2024-05-06T14:22:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying crash service with default placement... 2024-05-06T14:22:55.190 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:54 smithi153 ceph-mon[27177]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:55.190 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:54 smithi153 ceph-mon[27177]: Saving service mgr spec with placement count:2 2024-05-06T14:22:55.191 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:54 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f823d544700 1 Processor -- start 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f823d544700 1 -- start start 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f823d544700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238104e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f823d544700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8238105380 con 0x7f8238104a30 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f8236ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238104e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:55.449 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f8236ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238104e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57578/0 (socket says 172.21.15.153:57578) 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f8236ffd700 1 -- 172.21.15.153:0/1520883446 learned_addr learned my addr 172.21.15.153:0/1520883446 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.423+0000 7f8236ffd700 1 -- 172.21.15.153:0/1520883446 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82381054c0 con 0x7f8238104a30 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f8236ffd700 1 --2- 172.21.15.153:0/1520883446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238104e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82280098a0 tx=0x7f8228009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6369240609d529f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f8235ffb700 1 -- 172.21.15.153:0/1520883446 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8228004180 con 0x7f8238104a30 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f8235ffb700 1 -- 172.21.15.153:0/1520883446 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f82280042e0 con 0x7f8238104a30 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f8235ffb700 1 -- 172.21.15.153:0/1520883446 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8228011af0 con 0x7f8238104a30 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 -- 172.21.15.153:0/1520883446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 msgr2=0x7f8238104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 --2- 172.21.15.153:0/1520883446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238104e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82280098a0 tx=0x7f8228009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 -- 172.21.15.153:0/1520883446 shutdown_connections 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 --2- 172.21.15.153:0/1520883446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238104e40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 -- 172.21.15.153:0/1520883446 >> 172.21.15.153:0/1520883446 conn(0x7f8238100060 msgr2=0x7f82381024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 -- 172.21.15.153:0/1520883446 shutdown_connections 2024-05-06T14:22:55.450 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.424+0000 7f823d544700 1 -- 172.21.15.153:0/1520883446 wait complete. 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f823d544700 1 Processor -- start 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f823d544700 1 -- start start 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f823d544700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238079b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f823d544700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823819a430 con 0x7f8238104a30 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f8236ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238079b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f8236ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238079b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57582/0 (socket says 172.21.15.153:57582) 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.425+0000 7f8236ffd700 1 -- 172.21.15.153:0/3703790339 learned_addr learned my addr 172.21.15.153:0/3703790339 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f8236ffd700 1 -- 172.21.15.153:0/3703790339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228009550 con 0x7f8238104a30 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f8236ffd700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238079b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8228011670 tx=0x7f8228011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8228003a40 con 0x7f8238104a30 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823819ea40 con 0x7f8238104a30 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8228003ba0 con 0x7f8238104a30 2024-05-06T14:22:55.451 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8228024790 con 0x7f8238104a30 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.426+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823819eda0 con 0x7f8238104a30 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.427+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f822802d3f0 con 0x7f8238104a30 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.427+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8238062380 con 0x7f8238104a30 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.427+0000 7f821ffff700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f82200383f0 0x7f822003a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.427+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f822804c530 con 0x7f8238104a30 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.427+0000 7f82367fc700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f82200383f0 0x7f822003a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.427+0000 7f82367fc700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f82200383f0 0x7f822003a8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f822c006fd0 tx=0x7f822c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.432+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8228010970 con 0x7f8238104a30 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.653+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f82381023f0 con 0x7f82200383f0 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.657+0000 7f821ffff700 1 -- 172.21.15.153:0/3703790339 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f82381023f0 con 0x7f82200383f0 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f82200383f0 msgr2=0x7f822003a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f82200383f0 0x7f822003a8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f822c006fd0 tx=0x7f822c006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:55.452 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 msgr2=0x7f8238079b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238079b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8228011670 tx=0x7f8228011b50 comp rx=0 tx=0).stop 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 shutdown_connections 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f82200383f0 0x7f822003a8a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 --2- 172.21.15.153:0/3703790339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8238104a30 0x7f8238079b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 >> 172.21.15.153:0/3703790339 conn(0x7f8238100060 msgr2=0x7f82381017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 shutdown_connections 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:54.659+0000 7f823d544700 1 -- 172.21.15.153:0/3703790339 wait complete. 2024-05-06T14:22:55.453 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying ceph-exporter service with default placement... 2024-05-06T14:22:56.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:55 smithi153 ceph-mon[27177]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:56.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:55 smithi153 ceph-mon[27177]: Saving service crash spec with placement * 2024-05-06T14:22:56.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-05-06T14:22:56.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e9572700 1 Processor -- start 2024-05-06T14:22:56.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e9572700 1 -- start start 2024-05-06T14:22:56.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e9572700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e407a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:56.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e9572700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e407a850 con 0x7f56e407be10 2024-05-06T14:22:56.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e407a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:56.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e407a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57592/0 (socket says 172.21.15.153:57592) 2024-05-06T14:22:56.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e2ffd700 1 -- 172.21.15.153:0/2424665441 learned_addr learned my addr 172.21.15.153:0/2424665441 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.862+0000 7f56e2ffd700 1 -- 172.21.15.153:0/2424665441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e407a990 con 0x7f56e407be10 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e2ffd700 1 --2- 172.21.15.153:0/2424665441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e407a310 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56d40098a0 tx=0x7f56d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3429f3b88c0d27a server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e1ffb700 1 -- 172.21.15.153:0/2424665441 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d4004180 con 0x7f56e407be10 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e1ffb700 1 -- 172.21.15.153:0/2424665441 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56d40042e0 con 0x7f56e407be10 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e1ffb700 1 -- 172.21.15.153:0/2424665441 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d4011af0 con 0x7f56e407be10 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 -- 172.21.15.153:0/2424665441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 msgr2=0x7f56e407a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:56.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 --2- 172.21.15.153:0/2424665441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e407a310 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56d40098a0 tx=0x7f56d4009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 -- 172.21.15.153:0/2424665441 shutdown_connections 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 --2- 172.21.15.153:0/2424665441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e407a310 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 -- 172.21.15.153:0/2424665441 >> 172.21.15.153:0/2424665441 conn(0x7f56e4101180 msgr2=0x7f56e4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 -- 172.21.15.153:0/2424665441 shutdown_connections 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.863+0000 7f56e9572700 1 -- 172.21.15.153:0/2424665441 wait complete. 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.864+0000 7f56e9572700 1 Processor -- start 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.864+0000 7f56e9572700 1 -- start start 2024-05-06T14:22:56.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.864+0000 7f56e9572700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e419bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.864+0000 7f56e9572700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e419c010 con 0x7f56e407be10 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.864+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e419bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.864+0000 7f56e2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e419bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57598/0 (socket says 172.21.15.153:57598) 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56e2ffd700 1 -- 172.21.15.153:0/3210813102 learned_addr learned my addr 172.21.15.153:0/3210813102 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56e2ffd700 1 -- 172.21.15.153:0/3210813102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d4009550 con 0x7f56e407be10 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56e2ffd700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e419bad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f56d4000c00 tx=0x7f56d4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:56.781 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d4003a70 con 0x7f56e407be10 2024-05-06T14:22:56.782 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56d4003bd0 con 0x7f56e407be10 2024-05-06T14:22:56.782 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e419c210 con 0x7f56e407be10 2024-05-06T14:22:56.782 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d40238c0 con 0x7f56e407be10 2024-05-06T14:22:56.782 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.865+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e419c630 con 0x7f56e407be10 2024-05-06T14:22:56.782 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.866+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f56d4023b80 con 0x7f56e407be10 2024-05-06T14:22:56.782 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.866+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e4062380 con 0x7f56e407be10 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.866+0000 7f56cbfff700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f56cc0383f0 0x7f56cc03a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.866+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f56d404c400 con 0x7f56e407be10 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.866+0000 7f56e27fc700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f56cc0383f0 0x7f56cc03a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.867+0000 7f56e27fc700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f56cc0383f0 0x7f56cc03a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f56d8006fd0 tx=0x7f56d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:55.871+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f56d402b300 con 0x7f56e407be10 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.074+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f56e4103440 con 0x7f56cc0383f0 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.081+0000 7f56cbfff700 1 -- 172.21.15.153:0/3210813102 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f56e4103440 con 0x7f56cc0383f0 2024-05-06T14:22:56.783 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f56cc0383f0 msgr2=0x7f56cc03a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f56cc0383f0 0x7f56cc03a8a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f56d8006fd0 tx=0x7f56d8006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 msgr2=0x7f56e419bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e419bad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f56d4000c00 tx=0x7f56d4011b50 comp rx=0 tx=0).stop 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 shutdown_connections 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f56cc0383f0 0x7f56cc03a8a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 --2- 172.21.15.153:0/3210813102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56e407be10 0x7f56e419bad0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:56.784 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 >> 172.21.15.153:0/3210813102 conn(0x7f56e4101180 msgr2=0x7f56e4102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:56.785 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 shutdown_connections 2024-05-06T14:22:56.785 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:56.083+0000 7f56e9572700 1 -- 172.21.15.153:0/3210813102 wait complete. 2024-05-06T14:22:56.785 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying prometheus service with default placement... 2024-05-06T14:22:57.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:57 smithi153 ceph-mon[27177]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:57.160 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:57 smithi153 ceph-mon[27177]: Saving service ceph-exporter spec with placement * 2024-05-06T14:22:57.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:57 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:57.161 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:57 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.276+0000 7fc01f0ef700 1 Processor -- start 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.276+0000 7fc01f0ef700 1 -- start start 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.276+0000 7fc01f0ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc0181089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc01f0ef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc018074510 con 0x7fc018106610 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc01ce8b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc0181089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc01ce8b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc0181089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57604/0 (socket says 172.21.15.153:57604) 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc01ce8b700 1 -- 172.21.15.153:0/2846999370 learned_addr learned my addr 172.21.15.153:0/2846999370 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:58.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc01ce8b700 1 -- 172.21.15.153:0/2846999370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc018074650 con 0x7fc018106610 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc01ce8b700 1 --2- 172.21.15.153:0/2846999370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc0181089f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0098a0 tx=0x7fc00c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656651412b29700b server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc0177fe700 1 -- 172.21.15.153:0/2846999370 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c004180 con 0x7fc018106610 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc0177fe700 1 -- 172.21.15.153:0/2846999370 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc00c0042e0 con 0x7fc018106610 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.277+0000 7fc0177fe700 1 -- 172.21.15.153:0/2846999370 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c011af0 con 0x7fc018106610 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.278+0000 7fc01f0ef700 1 -- 172.21.15.153:0/2846999370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 msgr2=0x7fc0181089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.278+0000 7fc01f0ef700 1 --2- 172.21.15.153:0/2846999370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc0181089f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0098a0 tx=0x7fc00c009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.278+0000 7fc01f0ef700 1 -- 172.21.15.153:0/2846999370 shutdown_connections 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.278+0000 7fc01f0ef700 1 --2- 172.21.15.153:0/2846999370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc0181089f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.278+0000 7fc01f0ef700 1 -- 172.21.15.153:0/2846999370 >> 172.21.15.153:0/2846999370 conn(0x7fc018100070 msgr2=0x7fc018102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.278+0000 7fc01f0ef700 1 -- 172.21.15.153:0/2846999370 shutdown_connections 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.279+0000 7fc01f0ef700 1 -- 172.21.15.153:0/2846999370 wait complete. 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.279+0000 7fc01f0ef700 1 Processor -- start 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.279+0000 7fc01f0ef700 1 -- start start 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01f0ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc018199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:58.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01f0ef700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc018199dd0 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01ce8b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc018199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01ce8b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc018199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57614/0 (socket says 172.21.15.153:57614) 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01ce8b700 1 -- 172.21.15.153:0/1603517399 learned_addr learned my addr 172.21.15.153:0/1603517399 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01ce8b700 1 -- 172.21.15.153:0/1603517399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c009550 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01ce8b700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc018199890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc00c006ba0 tx=0x7fc00c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c003b00 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc00c01a8e0 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc018199fd0 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.280+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c019990 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.281+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01819a3f0 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.282+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc018062380 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.282+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fc00c019c00 con 0x7fc018106610 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.282+0000 7fc015ffb700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fc0080383f0 0x7fc00803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:58.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.282+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc00c04c660 con 0x7fc018106610 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.282+0000 7fc017fff700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fc0080383f0 0x7fc00803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.282+0000 7fc017fff700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fc0080383f0 0x7fc00803a8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc004006fd0 tx=0x7fc004006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.287+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc00c02d440 con 0x7fc018106610 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.508+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc018102390 con 0x7fc0080383f0 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.513+0000 7fc015ffb700 1 -- 172.21.15.153:0/1603517399 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc018102390 con 0x7fc0080383f0 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fc0080383f0 msgr2=0x7fc00803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fc0080383f0 0x7fc00803a8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc004006fd0 tx=0x7fc004006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 msgr2=0x7fc018199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc018199890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc00c006ba0 tx=0x7fc00c011b50 comp rx=0 tx=0).stop 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 shutdown_connections 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fc0080383f0 0x7fc00803a8a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 --2- 172.21.15.153:0/1603517399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc018106610 0x7fc018199890 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 >> 172.21.15.153:0/1603517399 conn(0x7fc018100070 msgr2=0x7fc018101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 shutdown_connections 2024-05-06T14:22:58.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:57.516+0000 7fc01f0ef700 1 -- 172.21.15.153:0/1603517399 wait complete. 2024-05-06T14:22:58.185 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying grafana service with default placement... 2024-05-06T14:22:58.424 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:58 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:58.424 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:58 smithi153 ceph-mon[27177]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:22:58.424 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:58 smithi153 ceph-mon[27177]: Saving service prometheus spec with placement count:1 2024-05-06T14:22:58.424 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:58 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:22:59.792 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-05-06T14:22:59.793 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f288b94e700 1 Processor -- start 2024-05-06T14:22:59.793 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f288b94e700 1 -- start start 2024-05-06T14:22:59.793 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f288b94e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f2884105f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:59.793 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f288b94e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884106460 con 0x7f2884105b10 2024-05-06T14:22:59.793 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f28896ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f2884105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:59.793 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f28896ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f2884105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57626/0 (socket says 172.21.15.153:57626) 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.671+0000 7f28896ea700 1 -- 172.21.15.153:0/4143313721 learned_addr learned my addr 172.21.15.153:0/4143313721 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f28896ea700 1 -- 172.21.15.153:0/4143313721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28841065a0 con 0x7f2884105b10 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f28896ea700 1 --2- 172.21.15.153:0/4143313721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f2884105f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28800098a0 tx=0x7f2880009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab7e290321a8e0d server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f287bfff700 1 -- 172.21.15.153:0/4143313721 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2880004180 con 0x7f2884105b10 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f287bfff700 1 -- 172.21.15.153:0/4143313721 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f28800042e0 con 0x7f2884105b10 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f287bfff700 1 -- 172.21.15.153:0/4143313721 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2880011af0 con 0x7f2884105b10 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f288b94e700 1 -- 172.21.15.153:0/4143313721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 msgr2=0x7f2884105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:59.794 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.672+0000 7f288b94e700 1 --2- 172.21.15.153:0/4143313721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f2884105f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f28800098a0 tx=0x7f2880009bb0 comp rx=0 tx=0).stop 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 -- 172.21.15.153:0/4143313721 shutdown_connections 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 --2- 172.21.15.153:0/4143313721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f2884105f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 -- 172.21.15.153:0/4143313721 >> 172.21.15.153:0/4143313721 conn(0x7f2884101180 msgr2=0x7f2884103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 -- 172.21.15.153:0/4143313721 shutdown_connections 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 -- 172.21.15.153:0/4143313721 wait complete. 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 Processor -- start 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 -- start start 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f288407bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.673+0000 7f288b94e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288407c180 con 0x7f2884105b10 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f28896ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f288407bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:59.795 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f28896ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f288407bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57640/0 (socket says 172.21.15.153:57640) 2024-05-06T14:22:59.796 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f28896ea700 1 -- 172.21.15.153:0/1717232007 learned_addr learned my addr 172.21.15.153:0/1717232007 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:22:59.796 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f28896ea700 1 -- 172.21.15.153:0/1717232007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2880009550 con 0x7f2884105b10 2024-05-06T14:22:59.796 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f28896ea700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f288407bc40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2880006ba0 tx=0x7f2880011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:59.796 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2880003b00 con 0x7f2884105b10 2024-05-06T14:22:59.796 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f288001a8e0 con 0x7f2884105b10 2024-05-06T14:22:59.796 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288407a290 con 0x7f2884105b10 2024-05-06T14:22:59.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28800199c0 con 0x7f2884105b10 2024-05-06T14:22:59.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.674+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288407a6b0 con 0x7f2884105b10 2024-05-06T14:22:59.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.675+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f2880019c80 con 0x7f2884105b10 2024-05-06T14:22:59.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.675+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2884062380 con 0x7f2884105b10 2024-05-06T14:22:59.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.675+0000 7f287a7fc700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f28700383a0 0x7f287003a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:22:59.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.675+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f288004c5b0 con 0x7f2884105b10 2024-05-06T14:22:59.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.675+0000 7f2888ee9700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f28700383a0 0x7f287003a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:22:59.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.676+0000 7f2888ee9700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f28700383a0 0x7f287003a850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2874006fd0 tx=0x7f2874006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:22:59.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.680+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2880018530 con 0x7f2884105b10 2024-05-06T14:22:59.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.885+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f2884102670 con 0x7f28700383a0 2024-05-06T14:22:59.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.890+0000 7f287a7fc700 1 -- 172.21.15.153:0/1717232007 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2884102670 con 0x7f28700383a0 2024-05-06T14:22:59.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.892+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f28700383a0 msgr2=0x7f287003a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.892+0000 7f288b94e700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f28700383a0 0x7f287003a850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2874006fd0 tx=0x7f2874006e40 comp rx=0 tx=0).stop 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 msgr2=0x7f288407bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f288407bc40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2880006ba0 tx=0x7f2880011b50 comp rx=0 tx=0).stop 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 shutdown_connections 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f28700383a0 0x7f287003a850 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 --2- 172.21.15.153:0/1717232007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2884105b10 0x7f288407bc40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 >> 172.21.15.153:0/1717232007 conn(0x7f2884101180 msgr2=0x7f2884101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:22:59.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 shutdown_connections 2024-05-06T14:22:59.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:22:58.893+0000 7f288b94e700 1 -- 172.21.15.153:0/1717232007 wait complete. 2024-05-06T14:22:59.800 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying node-exporter service with default placement... 2024-05-06T14:23:00.102 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:59 smithi153 ceph-mon[27177]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:00.103 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:59 smithi153 ceph-mon[27177]: Saving service grafana spec with placement count:1 2024-05-06T14:23:00.103 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:22:59 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-05-06T14:23:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.213+0000 7f9ba980c700 1 Processor -- start 2024-05-06T14:23:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.213+0000 7f9ba980c700 1 -- start start 2024-05-06T14:23:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba980c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4108a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba980c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4079d30 con 0x7f9ba4106640 2024-05-06T14:23:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4108a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4108a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54000/0 (socket says 172.21.15.153:54000) 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba2ffd700 1 -- 172.21.15.153:0/879956324 learned_addr learned my addr 172.21.15.153:0/879956324 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba2ffd700 1 -- 172.21.15.153:0/879956324 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba4079e70 con 0x7f9ba4106640 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba2ffd700 1 --2- 172.21.15.153:0/879956324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4108a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9b980098a0 tx=0x7f9b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46af35e9973d6119 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba1ffb700 1 -- 172.21.15.153:0/879956324 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98004180 con 0x7f9ba4106640 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba1ffb700 1 -- 172.21.15.153:0/879956324 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9b980042e0 con 0x7f9ba4106640 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.214+0000 7f9ba1ffb700 1 -- 172.21.15.153:0/879956324 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98011af0 con 0x7f9ba4106640 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 -- 172.21.15.153:0/879956324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 msgr2=0x7f9ba4108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 --2- 172.21.15.153:0/879956324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4108a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9b980098a0 tx=0x7f9b98009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 -- 172.21.15.153:0/879956324 shutdown_connections 2024-05-06T14:23:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 --2- 172.21.15.153:0/879956324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4108a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 -- 172.21.15.153:0/879956324 >> 172.21.15.153:0/879956324 conn(0x7f9ba4100080 msgr2=0x7f9ba4102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 -- 172.21.15.153:0/879956324 shutdown_connections 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.215+0000 7f9ba980c700 1 -- 172.21.15.153:0/879956324 wait complete. 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba980c700 1 Processor -- start 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba980c700 1 -- start start 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba980c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba980c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba41064e0 con 0x7f9ba4106640 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54012/0 (socket says 172.21.15.153:54012) 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba2ffd700 1 -- 172.21.15.153:0/814964070 learned_addr learned my addr 172.21.15.153:0/814964070 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba2ffd700 1 -- 172.21.15.153:0/814964070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98009550 con 0x7f9ba4106640 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.216+0000 7f9ba2ffd700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4105fa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b98006ba0 tx=0x7f9b98011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.217+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98003b00 con 0x7f9ba4106640 2024-05-06T14:23:01.071 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.217+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9b9801a8e0 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.217+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba4104650 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.217+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98019990 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.217+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba4104ad0 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.217+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f9b98019c50 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.218+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba4109b60 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.218+0000 7f9ba880a700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9b8c0383a0 0x7f9b8c03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.218+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9b9804c5a0 con 0x7f9ba4106640 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.218+0000 7f9ba27fc700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9b8c0383a0 0x7f9b8c03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.218+0000 7f9ba27fc700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9b8c0383a0 0x7f9b8c03a850 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9b94006fd0 tx=0x7f9b94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:01.072 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.222+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9b9802b700 con 0x7f9ba4106640 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.427+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ba4102380 con 0x7f9b8c0383a0 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.432+0000 7f9ba880a700 1 -- 172.21.15.153:0/814964070 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f9ba4102380 con 0x7f9b8c0383a0 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9b8c0383a0 msgr2=0x7f9b8c03a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9b8c0383a0 0x7f9b8c03a850 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9b94006fd0 tx=0x7f9b94006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 msgr2=0x7f9ba4105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4105fa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b98006ba0 tx=0x7f9b98011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 shutdown_connections 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f9b8c0383a0 0x7f9b8c03a850 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 --2- 172.21.15.153:0/814964070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba4106640 0x7f9ba4105fa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 >> 172.21.15.153:0/814964070 conn(0x7f9ba4100080 msgr2=0x7f9ba4101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 shutdown_connections 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:00.434+0000 7f9ba980c700 1 -- 172.21.15.153:0/814964070 wait complete. 2024-05-06T14:23:01.073 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying alertmanager service with default placement... 2024-05-06T14:23:01.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:01 smithi153 ceph-mon[27177]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:01.697 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:01 smithi153 ceph-mon[27177]: Saving service node-exporter spec with placement * 2024-05-06T14:23:01.697 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:01 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:01.697 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:01 smithi153 ceph-mon[27177]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.498+0000 7ff4944fb700 1 Processor -- start 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.498+0000 7ff4944fb700 1 -- start start 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.498+0000 7ff4944fb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.498+0000 7ff4944fb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c104750 con 0x7ff48c103e00 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff492297700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff492297700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54014/0 (socket says 172.21.15.153:54014) 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff492297700 1 -- 172.21.15.153:0/3138655103 learned_addr learned my addr 172.21.15.153:0/3138655103 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff492297700 1 -- 172.21.15.153:0/3138655103 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c104890 con 0x7ff48c103e00 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff492297700 1 --2- 172.21.15.153:0/3138655103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c104210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0098a0 tx=0x7ff47c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f757af45de0d4c15 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff491295700 1 -- 172.21.15.153:0/3138655103 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c004180 con 0x7ff48c103e00 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff491295700 1 -- 172.21.15.153:0/3138655103 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff47c0042e0 con 0x7ff48c103e00 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.499+0000 7ff491295700 1 -- 172.21.15.153:0/3138655103 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c011af0 con 0x7ff48c103e00 2024-05-06T14:23:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 -- 172.21.15.153:0/3138655103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 msgr2=0x7ff48c104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 --2- 172.21.15.153:0/3138655103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c104210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0098a0 tx=0x7ff47c009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 -- 172.21.15.153:0/3138655103 shutdown_connections 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 --2- 172.21.15.153:0/3138655103 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c104210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 -- 172.21.15.153:0/3138655103 >> 172.21.15.153:0/3138655103 conn(0x7ff48c0ff7e0 msgr2=0x7ff48c101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 -- 172.21.15.153:0/3138655103 shutdown_connections 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.500+0000 7ff4944fb700 1 -- 172.21.15.153:0/3138655103 wait complete. 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4944fb700 1 Processor -- start 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4944fb700 1 -- start start 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4944fb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4944fb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c106cd0 con 0x7ff48c103e00 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff492297700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff492297700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54028/0 (socket says 172.21.15.153:54028) 2024-05-06T14:23:02.258 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff492297700 1 -- 172.21.15.153:0/106819397 learned_addr learned my addr 172.21.15.153:0/106819397 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff492297700 1 -- 172.21.15.153:0/106819397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c009550 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff492297700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c106790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff47c000c00 tx=0x7ff47c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c003a70 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff47c003bd0 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c104de0 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.501+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c0238c0 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.502+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c105200 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.502+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7ff47c023b80 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.502+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c062380 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.502+0000 7ff4837fe700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7ff4780383f0 0x7ff47803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.503+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff47c04c230 con 0x7ff48c103e00 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.503+0000 7ff491a96700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7ff4780383f0 0x7ff47803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.503+0000 7ff491a96700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7ff4780383f0 0x7ff47803a8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff488006fd0 tx=0x7ff488006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:02.259 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.507+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff47c0193c0 con 0x7ff48c103e00 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.708+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7ff48c101ab0 con 0x7ff4780383f0 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.713+0000 7ff4837fe700 1 -- 172.21.15.153:0/106819397 <== mgr.14120 v2:172.21.15.153:6800/585874296 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7ff48c101ab0 con 0x7ff4780383f0 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7ff4780383f0 msgr2=0x7ff47803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7ff4780383f0 0x7ff47803a8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff488006fd0 tx=0x7ff488006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 msgr2=0x7ff48c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c106790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff47c000c00 tx=0x7ff47c011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 shutdown_connections 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7ff4780383f0 0x7ff47803a8a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 --2- 172.21.15.153:0/106819397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff48c103e00 0x7ff48c106790 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 >> 172.21.15.153:0/106819397 conn(0x7ff48c0ff7e0 msgr2=0x7ff48c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 shutdown_connections 2024-05-06T14:23:02.260 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:01.716+0000 7ff4944fb700 1 -- 172.21.15.153:0/106819397 wait complete. 2024-05-06T14:23:02.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:02 smithi153 ceph-mon[27177]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:02.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:02 smithi153 ceph-mon[27177]: Saving service alertmanager spec with placement count:1 2024-05-06T14:23:02.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:02 smithi153 ceph-mon[27177]: from='mgr.14120 172.21.15.153:0/1554990227' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:03.413 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f0393392700 1 Processor -- start 2024-05-06T14:23:03.413 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f0393392700 1 -- start start 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f0393392700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c107580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f0393392700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c107ac0 con 0x7f038c107170 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f039112e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c107580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f039112e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c107580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54044/0 (socket says 172.21.15.153:54044) 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f039112e700 1 -- 172.21.15.153:0/2329966709 learned_addr learned my addr 172.21.15.153:0/2329966709 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.649+0000 7f039112e700 1 -- 172.21.15.153:0/2329966709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c107c00 con 0x7f038c107170 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f039112e700 1 --2- 172.21.15.153:0/2329966709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c107580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03880098a0 tx=0x7f0388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c585d34b93d2116c server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0383fff700 1 -- 172.21.15.153:0/2329966709 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0388004180 con 0x7f038c107170 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0383fff700 1 -- 172.21.15.153:0/2329966709 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f03880042e0 con 0x7f038c107170 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0383fff700 1 -- 172.21.15.153:0/2329966709 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0388011af0 con 0x7f038c107170 2024-05-06T14:23:03.414 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0393392700 1 -- 172.21.15.153:0/2329966709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 msgr2=0x7f038c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0393392700 1 --2- 172.21.15.153:0/2329966709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c107580 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03880098a0 tx=0x7f0388009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0393392700 1 -- 172.21.15.153:0/2329966709 shutdown_connections 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0393392700 1 --2- 172.21.15.153:0/2329966709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c107580 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.650+0000 7f0393392700 1 -- 172.21.15.153:0/2329966709 >> 172.21.15.153:0/2329966709 conn(0x7f038c0ff7e0 msgr2=0x7f038c101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f0393392700 1 -- 172.21.15.153:0/2329966709 shutdown_connections 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f0393392700 1 -- 172.21.15.153:0/2329966709 wait complete. 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f0393392700 1 Processor -- start 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f0393392700 1 -- start start 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f0393392700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c106870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f0393392700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c106db0 con 0x7f038c107170 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f039112e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:03.415 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f039112e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54046/0 (socket says 172.21.15.153:54046) 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.651+0000 7f039112e700 1 -- 172.21.15.153:0/3271833917 learned_addr learned my addr 172.21.15.153:0/3271833917 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f039112e700 1 -- 172.21.15.153:0/3271833917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388009550 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f039112e700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c106870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0388006ba0 tx=0x7f0388011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0388003b00 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f038801a8e0 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038c104ec0 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03880199c0 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.652+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038c1052e0 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.653+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7f0388019c30 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.653+0000 7f03827fc700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f03780383f0 0x7f037803a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.653+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038c1046c0 con 0x7f038c107170 2024-05-06T14:23:03.416 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.653+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f038804c5a0 con 0x7f038c107170 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.653+0000 7f039092d700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f03780383f0 0x7f037803a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.653+0000 7f039092d700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f03780383f0 0x7f037803a8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f037c006fd0 tx=0x7f037c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.658+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0388018530 con 0x7f038c107170 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.846+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f038c105b80 con 0x7f038c107170 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.850+0000 7f03827fc700 1 -- 172.21.15.153:0/3271833917 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0388018350 con 0x7f038c107170 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f03780383f0 msgr2=0x7f037803a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f03780383f0 0x7f037803a8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f037c006fd0 tx=0x7f037c006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 msgr2=0x7f038c106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c106870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0388006ba0 tx=0x7f0388011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 shutdown_connections 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f03780383f0 0x7f037803a8a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 --2- 172.21.15.153:0/3271833917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f038c107170 0x7f038c106870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:03.417 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.852+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 >> 172.21.15.153:0/3271833917 conn(0x7f038c0ff7e0 msgr2=0x7f038c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:03.418 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.853+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 shutdown_connections 2024-05-06T14:23:03.418 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:02.853+0000 7f0393392700 1 -- 172.21.15.153:0/3271833917 wait complete. 2024-05-06T14:23:04.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:03 smithi153 ceph-mon[27177]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:04.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:03 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3271833917' entity='client.admin' 2024-05-06T14:23:04.651 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce85f0700 1 Processor -- start 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce85f0700 1 -- start start 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce85f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce0104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce85f0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0104750 con 0x7efce0103e00 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce638c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce0104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce638c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce0104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54050/0 (socket says 172.21.15.153:54050) 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce638c700 1 -- 172.21.15.153:0/1984651219 learned_addr learned my addr 172.21.15.153:0/1984651219 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.871+0000 7efce638c700 1 -- 172.21.15.153:0/1984651219 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0104890 con 0x7efce0103e00 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce638c700 1 --2- 172.21.15.153:0/1984651219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce0104210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0098a0 tx=0x7efcdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f97e360bb4373a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce538a700 1 -- 172.21.15.153:0/1984651219 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcdc004180 con 0x7efce0103e00 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce538a700 1 -- 172.21.15.153:0/1984651219 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efcdc0042e0 con 0x7efce0103e00 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce538a700 1 -- 172.21.15.153:0/1984651219 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcdc011af0 con 0x7efce0103e00 2024-05-06T14:23:04.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce85f0700 1 -- 172.21.15.153:0/1984651219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 msgr2=0x7efce0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce85f0700 1 --2- 172.21.15.153:0/1984651219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce0104210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0098a0 tx=0x7efcdc009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce85f0700 1 -- 172.21.15.153:0/1984651219 shutdown_connections 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce85f0700 1 --2- 172.21.15.153:0/1984651219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce0104210 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.872+0000 7efce85f0700 1 -- 172.21.15.153:0/1984651219 >> 172.21.15.153:0/1984651219 conn(0x7efce00ff7e0 msgr2=0x7efce0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce85f0700 1 -- 172.21.15.153:0/1984651219 shutdown_connections 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce85f0700 1 -- 172.21.15.153:0/1984651219 wait complete. 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce85f0700 1 Processor -- start 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce85f0700 1 -- start start 2024-05-06T14:23:04.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce85f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce01998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce85f0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0199e00 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce638c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce01998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce638c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce01998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54060/0 (socket says 172.21.15.153:54060) 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.873+0000 7efce638c700 1 -- 172.21.15.153:0/426889559 learned_addr learned my addr 172.21.15.153:0/426889559 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efce638c700 1 -- 172.21.15.153:0/426889559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc009550 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efce638c700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce01998c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efcdc006ba0 tx=0x7efcdc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcdc003b00 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efcdc01a8e0 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce019a000 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcdc019990 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.874+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce019a420 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.875+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7efcdc019c50 con 0x7efce0103e00 2024-05-06T14:23:04.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.875+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce0062380 con 0x7efce0103e00 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.875+0000 7efcd77fe700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7efccc0383a0 0x7efccc03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.875+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efcdc04c580 con 0x7efce0103e00 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.875+0000 7efce5b8b700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7efccc0383a0 0x7efccc03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.876+0000 7efce5b8b700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7efccc0383a0 0x7efccc03a850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efcd0006fd0 tx=0x7efcd0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:03.880+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efcdc018b40 con 0x7efce0103e00 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.072+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7efce0107a30 con 0x7efce0103e00 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.075+0000 7efcd77fe700 1 -- 172.21.15.153:0/426889559 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7efcdc018b40 con 0x7efce0103e00 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7efccc0383a0 msgr2=0x7efccc03a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7efccc0383a0 0x7efccc03a850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efcd0006fd0 tx=0x7efcd0006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 msgr2=0x7efce01998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce01998c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efcdc006ba0 tx=0x7efcdc011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 shutdown_connections 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7efccc0383a0 0x7efccc03a850 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:04.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 --2- 172.21.15.153:0/426889559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce0103e00 0x7efce01998c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:04.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.077+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 >> 172.21.15.153:0/426889559 conn(0x7efce00ff7e0 msgr2=0x7efce0100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:04.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.078+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 shutdown_connections 2024-05-06T14:23:04.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:04.078+0000 7efce85f0700 1 -- 172.21.15.153:0/426889559 wait complete. 2024-05-06T14:23:04.656 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling the dashboard module... 2024-05-06T14:23:05.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:05 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/426889559' entity='client.admin' 2024-05-06T14:23:05.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:05 smithi153 ceph-mon[27177]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:06.383 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:06 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2286360230' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61f351700 1 Processor -- start 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61f351700 1 -- start start 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61f351700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61f351700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618105860 con 0x7fa618104f10 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61d0ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61d0ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54066/0 (socket says 172.21.15.153:54066) 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.051+0000 7fa61d0ed700 1 -- 172.21.15.153:0/4273377158 learned_addr learned my addr 172.21.15.153:0/4273377158 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:06.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61d0ed700 1 -- 172.21.15.153:0/4273377158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6181059a0 con 0x7fa618104f10 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61d0ed700 1 --2- 172.21.15.153:0/4273377158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618105320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6140098a0 tx=0x7fa614009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7e45fb8e53b3f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa60ffff700 1 -- 172.21.15.153:0/4273377158 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa614004180 con 0x7fa618104f10 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa60ffff700 1 -- 172.21.15.153:0/4273377158 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa6140042e0 con 0x7fa618104f10 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa60ffff700 1 -- 172.21.15.153:0/4273377158 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa614011af0 con 0x7fa618104f10 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61f351700 1 -- 172.21.15.153:0/4273377158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 msgr2=0x7fa618105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61f351700 1 --2- 172.21.15.153:0/4273377158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618105320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6140098a0 tx=0x7fa614009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61f351700 1 -- 172.21.15.153:0/4273377158 shutdown_connections 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61f351700 1 --2- 172.21.15.153:0/4273377158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618105320 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.052+0000 7fa61f351700 1 -- 172.21.15.153:0/4273377158 >> 172.21.15.153:0/4273377158 conn(0x7fa6180ff7e0 msgr2=0x7fa618101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61f351700 1 -- 172.21.15.153:0/4273377158 shutdown_connections 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61f351700 1 -- 172.21.15.153:0/4273377158 wait complete. 2024-05-06T14:23:06.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61f351700 1 Processor -- start 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61f351700 1 -- start start 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61f351700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61f351700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618199dd0 con 0x7fa618104f10 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61d0ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61d0ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54078/0 (socket says 172.21.15.153:54078) 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.053+0000 7fa61d0ed700 1 -- 172.21.15.153:0/2286360230 learned_addr learned my addr 172.21.15.153:0/2286360230 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa61d0ed700 1 -- 172.21.15.153:0/2286360230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa614009550 con 0x7fa618104f10 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa61d0ed700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618199890 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa614000c00 tx=0x7fa614011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa614003a70 con 0x7fa618104f10 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa618199fd0 con 0x7fa618104f10 2024-05-06T14:23:06.695 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa614003bd0 con 0x7fa618104f10 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6140238c0 con 0x7fa618104f10 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.054+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61819a3f0 con 0x7fa618104f10 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.055+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 45084+0+0 (secure 0 0 0) 0x7fa614023b80 con 0x7fa618104f10 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.055+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa618062380 con 0x7fa618104f10 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.055+0000 7fa60e7fc700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa6040383f0 0x7fa60403a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.055+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa61404c390 con 0x7fa618104f10 2024-05-06T14:23:06.696 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.055+0000 7fa61c8ec700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa6040383f0 0x7fa60403a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.055+0000 7fa61c8ec700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa6040383f0 0x7fa60403a8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa608006fd0 tx=0x7fa608006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.060+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa6140109d0 con 0x7fa618104f10 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:05.303+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa61819d590 con 0x7fa618104f10 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.080+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7fa614010dc0 con 0x7fa618104f10 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.080+0000 7fa60e7fc700 1 -- 172.21.15.153:0/2286360230 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa614018350 con 0x7fa618104f10 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa6040383f0 msgr2=0x7fa60403a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa6040383f0 0x7fa60403a8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa608006fd0 tx=0x7fa608006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 msgr2=0x7fa618199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618199890 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa614000c00 tx=0x7fa614011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 shutdown_connections 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa6040383f0 0x7fa60403a8a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 --2- 172.21.15.153:0/2286360230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa618104f10 0x7fa618199890 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 >> 172.21.15.153:0/2286360230 conn(0x7fa6180ff7e0 msgr2=0x7fa618104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:06.697 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 shutdown_connections 2024-05-06T14:23:06.698 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:06.083+0000 7fa61f351700 1 -- 172.21.15.153:0/2286360230 wait complete. 2024-05-06T14:23:07.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:07 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2286360230' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-06T14:23:07.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:07 smithi153 ceph-mon[27177]: mgrmap e9: smithi153.dolfed(active, since 25s) 2024-05-06T14:23:08.000 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.dolfed", 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa08a86700 1 Processor -- start 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa08a86700 1 -- start start 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa08a86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa04108a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa08a86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa04079d30 con 0x7faa04106680 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa0259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa04108a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:08.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa0259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa04108a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54122/0 (socket says 172.21.15.153:54122) 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.135+0000 7faa0259c700 1 -- 172.21.15.153:0/2428482096 learned_addr learned my addr 172.21.15.153:0/2428482096 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.136+0000 7faa0259c700 1 -- 172.21.15.153:0/2428482096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa04079e70 con 0x7faa04106680 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.136+0000 7faa0259c700 1 --2- 172.21.15.153:0/2428482096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa04108a60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40098a0 tx=0x7fa9f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e139c9cbad1037b server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.136+0000 7faa0159a700 1 -- 172.21.15.153:0/2428482096 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4004180 con 0x7faa04106680 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.136+0000 7faa0159a700 1 -- 172.21.15.153:0/2428482096 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa9f40042e0 con 0x7faa04106680 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.136+0000 7faa0159a700 1 -- 172.21.15.153:0/2428482096 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4011af0 con 0x7faa04106680 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 -- 172.21.15.153:0/2428482096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 msgr2=0x7faa04108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 --2- 172.21.15.153:0/2428482096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa04108a60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40098a0 tx=0x7fa9f4009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 -- 172.21.15.153:0/2428482096 shutdown_connections 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 --2- 172.21.15.153:0/2428482096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa04108a60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:08.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 -- 172.21.15.153:0/2428482096 >> 172.21.15.153:0/2428482096 conn(0x7faa04100060 msgr2=0x7faa041024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 -- 172.21.15.153:0/2428482096 shutdown_connections 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.137+0000 7faa08a86700 1 -- 172.21.15.153:0/2428482096 wait complete. 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa08a86700 1 Processor -- start 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa08a86700 1 -- start start 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa08a86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa041998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa08a86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa04199de0 con 0x7faa04106680 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa0259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa041998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa0259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa041998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54126/0 (socket says 172.21.15.153:54126) 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa0259c700 1 -- 172.21.15.153:0/700366673 learned_addr learned my addr 172.21.15.153:0/700366673 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa0259c700 1 -- 172.21.15.153:0/700366673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4009550 con 0x7faa04106680 2024-05-06T14:23:08.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7faa0259c700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa041998a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4011670 tx=0x7fa9f4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.138+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f40039c0 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.139+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa9f4003b20 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.139+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0419a040 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.139+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f40247f0 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.139+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0419a4c0 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.139+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7fa9f4024ab0 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.140+0000 7fa9f37fe700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 0x7fa9ec03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.140+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa9f404c330 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.140+0000 7faa01d9b700 1 -- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 msgr2=0x7fa9ec03a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.140+0000 7faa01d9b700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 0x7fa9ec03a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.140+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa04109ba0 con 0x7faa04106680 2024-05-06T14:23:08.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.145+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa9f4022310 con 0x7faa04106680 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.341+0000 7faa01d9b700 1 -- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 msgr2=0x7fa9ec03a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.341+0000 7faa01d9b700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 0x7fa9ec03a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.433+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7faa0419d4c0 con 0x7faa04106680 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.433+0000 7fa9f37fe700 1 -- 172.21.15.153:0/700366673 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa9f4049030 con 0x7faa04106680 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 msgr2=0x7fa9ec03a8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 0x7fa9ec03a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 msgr2=0x7faa041998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa041998a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4011670 tx=0x7fa9f4011aa0 comp rx=0 tx=0).stop 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 shutdown_connections 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7fa9ec038400 0x7fa9ec03a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 --2- 172.21.15.153:0/700366673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa04106680 0x7faa041998a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:08.005 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.435+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 >> 172.21.15.153:0/700366673 conn(0x7faa04100060 msgr2=0x7faa041017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:08.006 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.436+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 shutdown_connections 2024-05-06T14:23:08.006 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:07.436+0000 7faa08a86700 1 -- 172.21.15.153:0/700366673 wait complete. 2024-05-06T14:23:08.006 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-05-06T14:23:08.006 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 9... 2024-05-06T14:23:08.244 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:08 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/700366673' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: Active manager daemon smithi153.dolfed restarted 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: Activating manager daemon smithi153.dolfed 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: osdmap e3: 0 total, 0 up, 0 in 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: mgrmap e10: smithi153.dolfed(active, starting, since 0.00349741s) 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr metadata", "who": "smithi153.dolfed", "id": "smithi153.dolfed"}]: dispatch 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: Manager daemon smithi153.dolfed is now available 2024-05-06T14:23:13.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:23:13.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:23:13.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/mirror_snapshot_schedule"}]: dispatch 2024-05-06T14:23:13.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:12 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/trash_purge_schedule"}]: dispatch 2024-05-06T14:23:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-05-06T14:23:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-06T14:23:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-06T14:23:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-05-06T14:23:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.457+0000 7f753ed0b700 1 Processor -- start 2024-05-06T14:23:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.457+0000 7f753ed0b700 1 -- start start 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.457+0000 7f753ed0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f7538108a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f753ed0b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7538079d30 con 0x7f7538106680 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f753caa7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f7538108a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f753caa7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f7538108a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54142/0 (socket says 172.21.15.153:54142) 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f753caa7700 1 -- 172.21.15.153:0/2390409385 learned_addr learned my addr 172.21.15.153:0/2390409385 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f753caa7700 1 -- 172.21.15.153:0/2390409385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7538079e70 con 0x7f7538106680 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f753caa7700 1 --2- 172.21.15.153:0/2390409385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f7538108a60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f752c0098a0 tx=0x7f752c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac1d5278c1c89ab server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f75377fe700 1 -- 172.21.15.153:0/2390409385 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c004180 con 0x7f7538106680 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f75377fe700 1 -- 172.21.15.153:0/2390409385 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f752c0042e0 con 0x7f7538106680 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.458+0000 7f75377fe700 1 -- 172.21.15.153:0/2390409385 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c011af0 con 0x7f7538106680 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 -- 172.21.15.153:0/2390409385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 msgr2=0x7f7538108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 --2- 172.21.15.153:0/2390409385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f7538108a60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f752c0098a0 tx=0x7f752c009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 -- 172.21.15.153:0/2390409385 shutdown_connections 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 --2- 172.21.15.153:0/2390409385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f7538108a60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 -- 172.21.15.153:0/2390409385 >> 172.21.15.153:0/2390409385 conn(0x7f7538100060 msgr2=0x7f75381024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 -- 172.21.15.153:0/2390409385 shutdown_connections 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.459+0000 7f753ed0b700 1 -- 172.21.15.153:0/2390409385 wait complete. 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753ed0b700 1 Processor -- start 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753ed0b700 1 -- start start 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753ed0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f75381998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753ed0b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7538199de0 con 0x7f7538106680 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753caa7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f75381998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753caa7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f75381998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54154/0 (socket says 172.21.15.153:54154) 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753caa7700 1 -- 172.21.15.153:0/519087543 learned_addr learned my addr 172.21.15.153:0/519087543 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.460+0000 7f753caa7700 1 -- 172.21.15.153:0/519087543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c009550 con 0x7f7538106680 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.461+0000 7f753caa7700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f75381998a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f752c011670 tx=0x7f752c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:14.672 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.461+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c0039f0 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.461+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f752c003b50 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.461+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753819a040 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.461+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c0247c0 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.461+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753819a4c0 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.462+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f752c02d3f0 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.462+0000 7f7535ffb700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.462+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 --> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7538109ba0 con 0x7f7528038490 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.462+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f752c04c3b0 con 0x7f7538106680 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.462+0000 7f7537fff700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 msgr2=0x7f752803a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.462+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.663+0000 7f7537fff700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 msgr2=0x7f752803a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:08.663+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:09.064+0000 7f7537fff700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 msgr2=0x7f752803a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:14.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:09.064+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:09.865+0000 7f7537fff700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 msgr2=0x7f752803a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:09.865+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:11.466+0000 7f7537fff700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 msgr2=0x7f752803a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/585874296 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:11.466+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:12.781+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 10) v1 ==== 44864+0+0 (secure 0 0 0) 0x7f752c01ac00 con 0x7f7538106680 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:12.781+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 msgr2=0x7f752803a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:12.781+0000 7f7535ffb700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.782+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f752c04df50 con 0x7f7538106680 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.783+0000 7f7535ffb700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f752803bd40 0x7f752803e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.783+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7538109ba0 con 0x7f752803bd40 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.783+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f752803bd40 0x7f752803e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:14.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.783+0000 7f7537fff700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f752803bd40 0x7f752803e120 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7524003a10 tx=0x7f7524009270 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.784+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f7538109ba0 con 0x7f752803bd40 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.790+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f753819abe0 con 0x7f752803bd40 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.790+0000 7f7535ffb700 1 -- 172.21.15.153:0/519087543 <== mgr.14164 v2:172.21.15.153:6800/2228103355 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f753819abe0 con 0x7f752803bd40 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f752803bd40 msgr2=0x7f752803e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f752803bd40 0x7f752803e120 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7524003a10 tx=0x7f7524009270 comp rx=0 tx=0).stop 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 msgr2=0x7f75381998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f75381998a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f752c011670 tx=0x7f752c011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 shutdown_connections 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f752803bd40 0x7f752803e120 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7538106680 0x7f75381998a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 --2- 172.21.15.153:0/519087543 >> [v2:172.21.15.153:6800/585874296,v1:172.21.15.153:6801/585874296] conn(0x7f7528038490 0x7f752803a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 >> 172.21.15.153:0/519087543 conn(0x7f7538100060 msgr2=0x7f75381017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:14.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 shutdown_connections 2024-05-06T14:23:14.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:13.791+0000 7f753ed0b700 1 -- 172.21.15.153:0/519087543 wait complete. 2024-05-06T14:23:14.676 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 9 is available 2024-05-06T14:23:14.676 INFO:teuthology.orchestra.run.smithi153.stdout:Generating a dashboard self-signed certificate... 2024-05-06T14:23:14.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:14 smithi153 ceph-mon[27177]: mgrmap e11: smithi153.dolfed(active, since 1.00543s) 2024-05-06T14:23:14.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:14 smithi153 ceph-mon[27177]: [06/May/2024:14:23:13] ENGINE Bus STARTING 2024-05-06T14:23:14.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:14 smithi153 ceph-mon[27177]: [06/May/2024:14:23:13] ENGINE Serving on https://172.21.15.153:7150 2024-05-06T14:23:14.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:14 smithi153 ceph-mon[27177]: [06/May/2024:14:23:14] ENGINE Serving on http://172.21.15.153:8765 2024-05-06T14:23:14.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:14 smithi153 ceph-mon[27177]: [06/May/2024:14:23:14] ENGINE Bus STARTED 2024-05-06T14:23:14.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:14 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:16.031 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.134+0000 7f252d340700 1 Processor -- start 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.134+0000 7f252d340700 1 -- start start 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f252d340700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f2528108a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f252d340700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2528079d10 con 0x7f2528106660 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2526ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f2528108a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2526ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f2528108a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47134/0 (socket says 172.21.15.153:47134) 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2526ffd700 1 -- 172.21.15.153:0/1161879830 learned_addr learned my addr 172.21.15.153:0/1161879830 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2526ffd700 1 -- 172.21.15.153:0/1161879830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528079e50 con 0x7f2528106660 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2526ffd700 1 --2- 172.21.15.153:0/1161879830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f2528108a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f25180098a0 tx=0x7f2518009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359457e865de1cf7 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2525ffb700 1 -- 172.21.15.153:0/1161879830 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2518004180 con 0x7f2528106660 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.135+0000 7f2525ffb700 1 -- 172.21.15.153:0/1161879830 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f25180042e0 con 0x7f2528106660 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f2525ffb700 1 -- 172.21.15.153:0/1161879830 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2518011af0 con 0x7f2528106660 2024-05-06T14:23:16.032 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 -- 172.21.15.153:0/1161879830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 msgr2=0x7f2528108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 --2- 172.21.15.153:0/1161879830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f2528108a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f25180098a0 tx=0x7f2518009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 -- 172.21.15.153:0/1161879830 shutdown_connections 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 --2- 172.21.15.153:0/1161879830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f2528108a40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 -- 172.21.15.153:0/1161879830 >> 172.21.15.153:0/1161879830 conn(0x7f2528100080 msgr2=0x7f2528102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 -- 172.21.15.153:0/1161879830 shutdown_connections 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.136+0000 7f252d340700 1 -- 172.21.15.153:0/1161879830 wait complete. 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.137+0000 7f252d340700 1 Processor -- start 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.137+0000 7f252d340700 1 -- start start 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.137+0000 7f252d340700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f252819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.137+0000 7f252d340700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252819bfd0 con 0x7f2528106660 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.137+0000 7f2526ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f252819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.137+0000 7f2526ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f252819ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47140/0 (socket says 172.21.15.153:47140) 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f2526ffd700 1 -- 172.21.15.153:0/3210841850 learned_addr learned my addr 172.21.15.153:0/3210841850 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f2526ffd700 1 -- 172.21.15.153:0/3210841850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518009550 con 0x7f2528106660 2024-05-06T14:23:16.033 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f2526ffd700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f252819ba90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2518006ba0 tx=0x7f2518011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2518003b00 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f251801a8e0 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2518019990 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252819c1d0 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.138+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252819c5f0 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.139+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 11) v1 ==== 44991+0+0 (secure 0 0 0) 0x7f2518019c50 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.139+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2528062380 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.139+0000 7f250ffff700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f25100382c0 0x7f251003a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.140+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f251804c530 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.140+0000 7f25267fc700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f25100382c0 0x7f251003a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.140+0000 7f25267fc700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f25100382c0 0x7f251003a770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f251c006fd0 tx=0x7f251c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.145+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f251802b700 con 0x7f2528106660 2024-05-06T14:23:16.034 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.182+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f251802c3a0 con 0x7f2528106660 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.352+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f2528102370 con 0x7f25100382c0 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.415+0000 7f250ffff700 1 -- 172.21.15.153:0/3210841850 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f2528102370 con 0x7f25100382c0 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f25100382c0 msgr2=0x7f251003a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f25100382c0 0x7f251003a770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f251c006fd0 tx=0x7f251c006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 msgr2=0x7f252819ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f252819ba90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2518006ba0 tx=0x7f2518011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 shutdown_connections 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f25100382c0 0x7f251003a770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 --2- 172.21.15.153:0/3210841850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2528106660 0x7f252819ba90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 >> 172.21.15.153:0/3210841850 conn(0x7f2528100080 msgr2=0x7f2528101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 shutdown_connections 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:15.417+0000 7f252d340700 1 -- 172.21.15.153:0/3210841850 wait complete. 2024-05-06T14:23:16.035 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial admin user... 2024-05-06T14:23:16.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:16 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:16.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:16 smithi153 ceph-mon[27177]: mgrmap e12: smithi153.dolfed(active, since 2s) 2024-05-06T14:23:16.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:16 smithi153 ceph-mon[27177]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:16.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:16 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:16.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:16 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:17 smithi153 ceph-mon[27177]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:17 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:18.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$kUmQjG1LPQdDd1S0T42ce.bTgZomL1VFd2l0KGXm9R2K8lCNMc05m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715005396, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.472+0000 7f6d27d7f700 1 Processor -- start 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.472+0000 7f6d27d7f700 1 -- start start 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.472+0000 7f6d27d7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 0x7f6d201089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d27d7f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20074510 con 0x7f6d20106610 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d25b1b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 0x7f6d201089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d25b1b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 0x7f6d201089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47152/0 (socket says 172.21.15.153:47152) 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d25b1b700 1 -- 172.21.15.153:0/3999216902 learned_addr learned my addr 172.21.15.153:0/3999216902 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d25b1b700 1 -- 172.21.15.153:0/3999216902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d20074650 con 0x7f6d20106610 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d25b1b700 1 --2- 172.21.15.153:0/3999216902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 0x7f6d201089f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d180098a0 tx=0x7f6d18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cacdcc95ef6721 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d24b19700 1 -- 172.21.15.153:0/3999216902 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d18004180 con 0x7f6d20106610 2024-05-06T14:23:18.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d24b19700 1 -- 172.21.15.153:0/3999216902 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d180042e0 con 0x7f6d20106610 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.473+0000 7f6d24b19700 1 -- 172.21.15.153:0/3999216902 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d18011af0 con 0x7f6d20106610 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 -- 172.21.15.153:0/3999216902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 msgr2=0x7f6d201089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 --2- 172.21.15.153:0/3999216902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 0x7f6d201089f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d180098a0 tx=0x7f6d18009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 -- 172.21.15.153:0/3999216902 shutdown_connections 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 --2- 172.21.15.153:0/3999216902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20106610 0x7f6d201089f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 -- 172.21.15.153:0/3999216902 >> 172.21.15.153:0/3999216902 conn(0x7f6d20100070 msgr2=0x7f6d20102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 -- 172.21.15.153:0/3999216902 shutdown_connections 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.474+0000 7f6d27d7f700 1 -- 172.21.15.153:0/3999216902 wait complete. 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d27d7f700 1 Processor -- start 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d27d7f700 1 -- start start 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d27d7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 0x7f6d20199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d27d7f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2019a3f0 con 0x7f6d20199aa0 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d25b1b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 0x7f6d20199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:18.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d25b1b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 0x7f6d20199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47164/0 (socket says 172.21.15.153:47164) 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d25b1b700 1 -- 172.21.15.153:0/638633811 learned_addr learned my addr 172.21.15.153:0/638633811 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d25b1b700 1 -- 172.21.15.153:0/638633811 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18009550 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d25b1b700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 0x7f6d20199eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009fb0 tx=0x7f6d18011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d18003a00 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d18003b60 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2019a5f0 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.475+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d1801b730 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.476+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2019d250 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.476+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f6d18021070 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.477+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d201939a0 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.477+0000 7f6d1effd700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f6d08038450 0x7f6d0803a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.477+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6d1804c970 con 0x7f6d20199aa0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.477+0000 7f6d2531a700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f6d08038450 0x7f6d0803a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:18.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.477+0000 7f6d2531a700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f6d08038450 0x7f6d0803a900 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.481+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6d1801aba0 con 0x7f6d20199aa0 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.696+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6d200621a0 con 0x7f6d08038450 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.919+0000 7f6d1effd700 1 -- 172.21.15.153:0/638633811 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6d200621a0 con 0x7f6d08038450 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f6d08038450 msgr2=0x7f6d0803a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f6d08038450 0x7f6d0803a900 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 msgr2=0x7f6d20199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 0x7f6d20199eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009fb0 tx=0x7f6d18011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:18.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 shutdown_connections 2024-05-06T14:23:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f6d08038450 0x7f6d0803a900 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 --2- 172.21.15.153:0/638633811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d20199aa0 0x7f6d20199eb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 >> 172.21.15.153:0/638633811 conn(0x7f6d20100070 msgr2=0x7f6d201017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.921+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 shutdown_connections 2024-05-06T14:23:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:16.922+0000 7f6d27d7f700 1 -- 172.21.15.153:0/638633811 wait complete. 2024-05-06T14:23:18.202 INFO:teuthology.orchestra.run.smithi153.stdout:Fetching dashboard port number... 2024-05-06T14:23:19.142 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:18 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2327801225' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 8443 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.617+0000 7f3e6999b700 1 Processor -- start 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.617+0000 7f3e6999b700 1 -- start start 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.617+0000 7f3e6999b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.617+0000 7f3e6999b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e64105860 con 0x7f3e64104f10 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47178/0 (socket says 172.21.15.153:47178) 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e62ffd700 1 -- 172.21.15.153:0/3657256213 learned_addr learned my addr 172.21.15.153:0/3657256213 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e62ffd700 1 -- 172.21.15.153:0/3657256213 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e641059a0 con 0x7f3e64104f10 2024-05-06T14:23:19.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e62ffd700 1 --2- 172.21.15.153:0/3657256213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64105320 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009e80 tx=0x7f3e5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5dcfe6c883bbc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e61ffb700 1 -- 172.21.15.153:0/3657256213 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e5400e070 con 0x7f3e64104f10 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e61ffb700 1 -- 172.21.15.153:0/3657256213 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e5400b8c0 con 0x7f3e64104f10 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.618+0000 7f3e61ffb700 1 -- 172.21.15.153:0/3657256213 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e540044d0 con 0x7f3e64104f10 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 -- 172.21.15.153:0/3657256213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 msgr2=0x7f3e64105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 --2- 172.21.15.153:0/3657256213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64105320 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009e80 tx=0x7f3e5400b330 comp rx=0 tx=0).stop 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 -- 172.21.15.153:0/3657256213 shutdown_connections 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 --2- 172.21.15.153:0/3657256213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64105320 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 -- 172.21.15.153:0/3657256213 >> 172.21.15.153:0/3657256213 conn(0x7f3e640ff7e0 msgr2=0x7f3e64101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 -- 172.21.15.153:0/3657256213 shutdown_connections 2024-05-06T14:23:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 -- 172.21.15.153:0/3657256213 wait complete. 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.619+0000 7f3e6999b700 1 Processor -- start 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e6999b700 1 -- start start 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e6999b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e6999b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e64199dc0 con 0x7f3e64104f10 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47190/0 (socket says 172.21.15.153:47190) 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e62ffd700 1 -- 172.21.15.153:0/2327801225 learned_addr learned my addr 172.21.15.153:0/2327801225 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e62ffd700 1 -- 172.21.15.153:0/2327801225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54009b30 con 0x7f3e64104f10 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e62ffd700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64199880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e540072a0 tx=0x7f3e540049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e5400e040 con 0x7f3e64104f10 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3e5400bd80 con 0x7f3e64104f10 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.620+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e64199fc0 con 0x7f3e64104f10 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.621+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e540126c0 con 0x7f3e64104f10 2024-05-06T14:23:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.621+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6419a440 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.621+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 45097+0+0 (secure 0 0 0) 0x7f3e54019040 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.621+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e64062380 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.622+0000 7f3e68999700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3e50038400 0x7f3e5003a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.622+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e5404c130 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.622+0000 7f3e627fc700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3e50038400 0x7f3e5003a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.622+0000 7f3e627fc700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3e50038400 0x7f3e5003a8b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.626+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3e54011750 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.814+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f3e6419d240 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.814+0000 7f3e68999700 1 -- 172.21.15.153:0/2327801225 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f3e5401f070 con 0x7f3e64104f10 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3e50038400 msgr2=0x7f3e5003a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3e50038400 0x7f3e5003a8b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 msgr2=0x7f3e64199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64199880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e540072a0 tx=0x7f3e540049f0 comp rx=0 tx=0).stop 2024-05-06T14:23:19.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 shutdown_connections 2024-05-06T14:23:19.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3e50038400 0x7f3e5003a8b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:19.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 --2- 172.21.15.153:0/2327801225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e64104f10 0x7f3e64199880 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:19.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 >> 172.21.15.153:0/2327801225 conn(0x7f3e640ff7e0 msgr2=0x7f3e64104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:19.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 shutdown_connections 2024-05-06T14:23:19.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:18.816+0000 7f3e6999b700 1 -- 172.21.15.153:0/2327801225 wait complete. 2024-05-06T14:23:19.407 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-06T14:23:19.407 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-05-06T14:23:19.414 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-06T14:23:19.414 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-05-06T14:23:19.414 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-05-06T14:23:19.414 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-06T14:23:19.415 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph Dashboard is now available at: 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: URL: https://smithi153:8443/ 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: User: admin 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: Password: xc267vjsip 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout:Saving cluster configuration to /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config directory 2024-05-06T14:23:19.416 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling autotune for osd_memory_target 2024-05-06T14:23:20.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:19 smithi153 ceph-mon[27177]: mgrmap e13: smithi153.dolfed(active, since 6s) 2024-05-06T14:23:20.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faefb03a700 1 Processor -- start 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faefb03a700 1 -- start start 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faefb03a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef4104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faefb03a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef4104750 con 0x7faef4103e00 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faef8dd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef4104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faef8dd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef4104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53032/0 (socket says 172.21.15.153:53032) 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faef8dd6700 1 -- 172.21.15.153:0/2187046026 learned_addr learned my addr 172.21.15.153:0/2187046026 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.844+0000 7faef8dd6700 1 -- 172.21.15.153:0/2187046026 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef4104890 con 0x7faef4103e00 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faef8dd6700 1 --2- 172.21.15.153:0/2187046026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef4104210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faee80098a0 tx=0x7faee8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35cbc6cc53a2d718 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faef37fe700 1 -- 172.21.15.153:0/2187046026 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faee8004180 con 0x7faef4103e00 2024-05-06T14:23:20.633 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faef37fe700 1 -- 172.21.15.153:0/2187046026 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faee80042e0 con 0x7faef4103e00 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faef37fe700 1 -- 172.21.15.153:0/2187046026 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faee8011af0 con 0x7faef4103e00 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 -- 172.21.15.153:0/2187046026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 msgr2=0x7faef4104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 --2- 172.21.15.153:0/2187046026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef4104210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faee80098a0 tx=0x7faee8009bb0 comp rx=0 tx=0).stop 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 -- 172.21.15.153:0/2187046026 shutdown_connections 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 --2- 172.21.15.153:0/2187046026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef4104210 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 -- 172.21.15.153:0/2187046026 >> 172.21.15.153:0/2187046026 conn(0x7faef40ff7e0 msgr2=0x7faef4101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 -- 172.21.15.153:0/2187046026 shutdown_connections 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.845+0000 7faefb03a700 1 -- 172.21.15.153:0/2187046026 wait complete. 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faefb03a700 1 Processor -- start 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faefb03a700 1 -- start start 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faefb03a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef41998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faefb03a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef4199e00 con 0x7faef4103e00 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faef8dd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef41998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:20.634 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faef8dd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef41998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53042/0 (socket says 172.21.15.153:53042) 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faef8dd6700 1 -- 172.21.15.153:0/1711581374 learned_addr learned my addr 172.21.15.153:0/1711581374 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.846+0000 7faef8dd6700 1 -- 172.21.15.153:0/1711581374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8009550 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.847+0000 7faef8dd6700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef41998c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faee8000c00 tx=0x7faee8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.847+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faee8003a70 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.847+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faee8003bd0 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.847+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef419a000 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.847+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faee80238c0 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.847+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef419a420 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.848+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7faee8023b80 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.848+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef4062380 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.848+0000 7faef1ffb700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faee40384e0 0x7faee403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.848+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faee804c6c0 con 0x7faef4103e00 2024-05-06T14:23:20.635 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.848+0000 7faef3fff700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faee40384e0 0x7faee403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.848+0000 7faef3fff700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faee40384e0 0x7faee403a990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faee0006fd0 tx=0x7faee0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:19.853+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7faee8010d10 con 0x7faef4103e00 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.045+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7faef419d2f0 con 0x7faef4103e00 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.045+0000 7faef1ffb700 1 -- 172.21.15.153:0/1711581374 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7faee8010b30 con 0x7faef4103e00 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faee40384e0 msgr2=0x7faee403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faee40384e0 0x7faee403a990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faee0006fd0 tx=0x7faee0006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 msgr2=0x7faef41998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef41998c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faee8000c00 tx=0x7faee8011b50 comp rx=0 tx=0).stop 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 shutdown_connections 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faee40384e0 0x7faee403a990 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 --2- 172.21.15.153:0/1711581374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faef4103e00 0x7faef41998c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 >> 172.21.15.153:0/1711581374 conn(0x7faef40ff7e0 msgr2=0x7faef4100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 shutdown_connections 2024-05-06T14:23:20.636 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:20.048+0000 7faefb03a700 1 -- 172.21.15.153:0/1711581374 wait complete. 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.098+0000 7fc2d019c700 1 Processor -- start 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.098+0000 7fc2d019c700 1 -- start start 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.098+0000 7fc2d019c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c8108a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.098+0000 7fc2d019c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8079d10 con 0x7fc2c81066a0 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2cdf38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c8108a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2cdf38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c8108a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53050/0 (socket says 172.21.15.153:53050) 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2cdf38700 1 -- 172.21.15.153:0/3801495562 learned_addr learned my addr 172.21.15.153:0/3801495562 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2cdf38700 1 -- 172.21.15.153:0/3801495562 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c8079e50 con 0x7fc2c81066a0 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2cdf38700 1 --2- 172.21.15.153:0/3801495562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c8108a80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009e80 tx=0x7fc2c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235077d46294c6cd server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:21.964 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2ccf36700 1 -- 172.21.15.153:0/3801495562 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c400e070 con 0x7fc2c81066a0 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2ccf36700 1 -- 172.21.15.153:0/3801495562 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc2c400b8c0 con 0x7fc2c81066a0 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.099+0000 7fc2ccf36700 1 -- 172.21.15.153:0/3801495562 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c40044d0 con 0x7fc2c81066a0 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 -- 172.21.15.153:0/3801495562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 msgr2=0x7fc2c8108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 --2- 172.21.15.153:0/3801495562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c8108a80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009e80 tx=0x7fc2c400b330 comp rx=0 tx=0).stop 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 -- 172.21.15.153:0/3801495562 shutdown_connections 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 --2- 172.21.15.153:0/3801495562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c8108a80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 -- 172.21.15.153:0/3801495562 >> 172.21.15.153:0/3801495562 conn(0x7fc2c8100060 msgr2=0x7fc2c81024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 -- 172.21.15.153:0/3801495562 shutdown_connections 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 -- 172.21.15.153:0/3801495562 wait complete. 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 Processor -- start 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.100+0000 7fc2d019c700 1 -- start start 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2d019c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c81998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2d019c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8199df0 con 0x7fc2c81066a0 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2cdf38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c81998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:21.965 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2cdf38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c81998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53056/0 (socket says 172.21.15.153:53056) 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2cdf38700 1 -- 172.21.15.153:0/1318381193 learned_addr learned my addr 172.21.15.153:0/1318381193 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2cdf38700 1 -- 172.21.15.153:0/1318381193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4009b30 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2cdf38700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c81998b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009e50 tx=0x7fc2c40049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c400e040 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc2c400bcc0 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c8199ff0 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c4012650 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.101+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c819a470 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.102+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7fc2c4019040 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.102+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c8062380 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.102+0000 7fc2beffd700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fc2b40384f0 0x7fc2b403a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.102+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc2c404c020 con 0x7fc2c81066a0 2024-05-06T14:23:21.966 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.102+0000 7fc2cd737700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fc2b40384f0 0x7fc2b403a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:21.967 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.103+0000 7fc2cd737700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fc2b40384f0 0x7fc2b403a9a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8006fd0 tx=0x7fc2b8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:21.967 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.107+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc2c407e720 con 0x7fc2c81066a0 2024-05-06T14:23:21.967 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.401+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fc2c819d300 con 0x7fc2c81066a0 2024-05-06T14:23:21.967 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.404+0000 7fc2beffd700 1 -- 172.21.15.153:0/1318381193 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7fc2c401f070 con 0x7fc2c81066a0 2024-05-06T14:23:21.967 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fc2b40384f0 msgr2=0x7fc2b403a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fc2b40384f0 0x7fc2b403a9a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8006fd0 tx=0x7fc2b8006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 msgr2=0x7fc2c81998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c81998b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4009e50 tx=0x7fc2c40049f0 comp rx=0 tx=0).stop 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 shutdown_connections 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fc2b40384f0 0x7fc2b403a9a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 --2- 172.21.15.153:0/1318381193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2c81066a0 0x7fc2c81998b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 >> 172.21.15.153:0/1318381193 conn(0x7fc2c8100060 msgr2=0x7fc2c8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 shutdown_connections 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-05-06T14:23:21.406+0000 7fc2d019c700 1 -- 172.21.15.153:0/1318381193 wait complete. 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:23:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout:Or, if you are only running a single cluster on this host: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: ceph telemetry on 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout:For more information see: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:21.969 INFO:teuthology.orchestra.run.smithi153.stdout:Bootstrap complete. 2024-05-06T14:23:21.993 INFO:tasks.cephadm:Fetching config... 2024-05-06T14:23:21.993 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:23:21.994 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-06T14:23:22.009 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-06T14:23:22.010 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:23:22.010 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-06T14:23:22.066 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-06T14:23:22.067 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:23:22.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/keyring of=/dev/stdout 2024-05-06T14:23:22.131 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-06T14:23:22.131 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:23:22.131 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-06T14:23:22.185 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-06T14:23:22.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS66Y9i5ffT8perfG2ahf3etinZhNNBXz4H0GeBuHxg+d2GoqSQfUcNATCCs8V/WjeEI4XUBxNo8ghN/L8RLlK+bN6/20Ab2fcfXGTHGDVO4faVnI+tMAhdTfwyuquNEc8BWKj4uZZTn77YhXlp8/ibSNZFAZbs8zSOHQpFKZUR3JyOiYIuLvB14KWXCAk75ZZ1IckF7TYWv7AUZTO+EIoisifBYQZcPDFjtLLNQObo9um7EtnXEhMCKao7QXk+G/8M0KpksUDEvxXbi2pPXPnCUHzpUWvP+stnDe5kNmqLIFR16HAzt1+bkKcEbckLNcULYZ+SmC92x62v4uDhfM2+Pr4lEsnF/M20L2V7nNrxjqZqYRTXxhNizSBHqOzdGQl4DPETLfw3oxifrjfVqr/yyyRZN8F9cLXWnDx1uR/pV1HDsRKeU23szG5SCV1LMTOw3FHP0ajho2HvkUoVsqlHBfJfc7btjxXBe7iL1hNtPwrpkpK846oP542Lgkl7L8= ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-06T14:23:22.258 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS66Y9i5ffT8perfG2ahf3etinZhNNBXz4H0GeBuHxg+d2GoqSQfUcNATCCs8V/WjeEI4XUBxNo8ghN/L8RLlK+bN6/20Ab2fcfXGTHGDVO4faVnI+tMAhdTfwyuquNEc8BWKj4uZZTn77YhXlp8/ibSNZFAZbs8zSOHQpFKZUR3JyOiYIuLvB14KWXCAk75ZZ1IckF7TYWv7AUZTO+EIoisifBYQZcPDFjtLLNQObo9um7EtnXEhMCKao7QXk+G/8M0KpksUDEvxXbi2pPXPnCUHzpUWvP+stnDe5kNmqLIFR16HAzt1+bkKcEbckLNcULYZ+SmC92x62v4uDhfM2+Pr4lEsnF/M20L2V7nNrxjqZqYRTXxhNizSBHqOzdGQl4DPETLfw3oxifrjfVqr/yyyRZN8F9cLXWnDx1uR/pV1HDsRKeU23szG5SCV1LMTOw3FHP0ajho2HvkUoVsqlHBfJfc7btjxXBe7iL1hNtPwrpkpK846oP542Lgkl7L8= ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:23:22.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS66Y9i5ffT8perfG2ahf3etinZhNNBXz4H0GeBuHxg+d2GoqSQfUcNATCCs8V/WjeEI4XUBxNo8ghN/L8RLlK+bN6/20Ab2fcfXGTHGDVO4faVnI+tMAhdTfwyuquNEc8BWKj4uZZTn77YhXlp8/ibSNZFAZbs8zSOHQpFKZUR3JyOiYIuLvB14KWXCAk75ZZ1IckF7TYWv7AUZTO+EIoisifBYQZcPDFjtLLNQObo9um7EtnXEhMCKao7QXk+G/8M0KpksUDEvxXbi2pPXPnCUHzpUWvP+stnDe5kNmqLIFR16HAzt1+bkKcEbckLNcULYZ+SmC92x62v4uDhfM2+Pr4lEsnF/M20L2V7nNrxjqZqYRTXxhNizSBHqOzdGQl4DPETLfw3oxifrjfVqr/yyyRZN8F9cLXWnDx1uR/pV1HDsRKeU23szG5SCV1LMTOw3FHP0ajho2HvkUoVsqlHBfJfc7btjxXBe7iL1hNtPwrpkpK846oP542Lgkl7L8= ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-06T14:23:22.307 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS66Y9i5ffT8perfG2ahf3etinZhNNBXz4H0GeBuHxg+d2GoqSQfUcNATCCs8V/WjeEI4XUBxNo8ghN/L8RLlK+bN6/20Ab2fcfXGTHGDVO4faVnI+tMAhdTfwyuquNEc8BWKj4uZZTn77YhXlp8/ibSNZFAZbs8zSOHQpFKZUR3JyOiYIuLvB14KWXCAk75ZZ1IckF7TYWv7AUZTO+EIoisifBYQZcPDFjtLLNQObo9um7EtnXEhMCKao7QXk+G/8M0KpksUDEvxXbi2pPXPnCUHzpUWvP+stnDe5kNmqLIFR16HAzt1+bkKcEbckLNcULYZ+SmC92x62v4uDhfM2+Pr4lEsnF/M20L2V7nNrxjqZqYRTXxhNizSBHqOzdGQl4DPETLfw3oxifrjfVqr/yyyRZN8F9cLXWnDx1uR/pV1HDsRKeU23szG5SCV1LMTOw3FHP0ajho2HvkUoVsqlHBfJfc7btjxXBe7iL1hNtPwrpkpK846oP542Lgkl7L8= ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:23:22.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-06T14:23:22.485 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:22 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1318381193' entity='client.admin' 2024-05-06T14:23:22.642 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:23:25.020 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:24 smithi153 ceph-mon[27177]: Deploying daemon ceph-exporter.smithi153 on smithi153 2024-05-06T14:23:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/1463634158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 msgr2=0x7f4f440ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 --2- 172.21.15.153:0/1463634158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440ff140 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009910 tx=0x7f4f40009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/1463634158 shutdown_connections 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 --2- 172.21.15.153:0/1463634158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440ff140 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/1463634158 >> 172.21.15.153:0/1463634158 conn(0x7f4f440fa2c0 msgr2=0x7f4f440fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/1463634158 shutdown_connections 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.089+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/1463634158 wait complete. 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f4b0c9700 1 Processor -- start 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f4b0c9700 1 -- start start 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f4b0c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440fc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f4b0c9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f440fc7b0 con 0x7f4f440fed30 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f48e65700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f48e65700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53082/0 (socket says 172.21.15.153:53082) 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f48e65700 1 -- 172.21.15.153:0/2317793681 learned_addr learned my addr 172.21.15.153:0/2317793681 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:25.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.090+0000 7f4f48e65700 1 -- 172.21.15.153:0/2317793681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f400095f0 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.091+0000 7f4f48e65700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440fc270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4f40005950 tx=0x7f4f40009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.091+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4001d070 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.091+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f440fa8c0 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.091+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f440fad60 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.091+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4f4000be10 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.091+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4000f460 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.092+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f4f4000f5c0 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.092+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f44062380 con 0x7f4f440fed30 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.092+0000 7f4f39ffb700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4f2c0384e0 0x7f4f2c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.092+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f4004c520 con 0x7f4f440fed30 2024-05-06T14:23:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.092+0000 7f4f3bfff700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4f2c0384e0 0x7f4f2c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.093+0000 7f4f3bfff700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4f2c0384e0 0x7f4f2c03a990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f34006fd0 tx=0x7f4f34006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:25.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.097+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4f400207e0 con 0x7f4f440fed30 2024-05-06T14:23:25.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.290+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4f4418e900 con 0x7f4f440fed30 2024-05-06T14:23:25.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.294+0000 7f4f39ffb700 1 -- 172.21.15.153:0/2317793681 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4f40027070 con 0x7f4f440fed30 2024-05-06T14:23:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4f2c0384e0 msgr2=0x7f4f2c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4f2c0384e0 0x7f4f2c03a990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f34006fd0 tx=0x7f4f34006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 msgr2=0x7f4f440fc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:25.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440fc270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4f40005950 tx=0x7f4f40009e60 comp rx=0 tx=0).stop 2024-05-06T14:23:25.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 shutdown_connections 2024-05-06T14:23:25.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4f2c0384e0 0x7f4f2c03a990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:25.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 --2- 172.21.15.153:0/2317793681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f440fed30 0x7f4f440fc270 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:25.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 >> 172.21.15.153:0/2317793681 conn(0x7f4f440fa2c0 msgr2=0x7f4f44069080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:25.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.296+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 shutdown_connections 2024-05-06T14:23:25.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:25.297+0000 7f4f4b0c9700 1 -- 172.21.15.153:0/2317793681 wait complete. 2024-05-06T14:23:26.120 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-06T14:23:26.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-06T14:23:26.393 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:26 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2317793681' entity='client.admin' 2024-05-06T14:23:27.146 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:23:28.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 -- 172.21.15.153:0/2806637897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 msgr2=0x7f54f8102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:28.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 --2- 172.21.15.153:0/2806637897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8102d00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009910 tx=0x7f54e8009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:28.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 -- 172.21.15.153:0/2806637897 shutdown_connections 2024-05-06T14:23:28.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 --2- 172.21.15.153:0/2806637897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8102d00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:28.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 -- 172.21.15.153:0/2806637897 >> 172.21.15.153:0/2806637897 conn(0x7f54f80fa2e0 msgr2=0x7f54f80fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 -- 172.21.15.153:0/2806637897 shutdown_connections 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 -- 172.21.15.153:0/2806637897 wait complete. 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.650+0000 7f54fd19b700 1 Processor -- start 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54fd19b700 1 -- start start 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54fd19b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54fd19b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f8195490 con 0x7f54f8100920 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54f6d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54f6d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8194f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53108/0 (socket says 172.21.15.153:53108) 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54f6d9d700 1 -- 172.21.15.153:0/2915832522 learned_addr learned my addr 172.21.15.153:0/2915832522 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54f6d9d700 1 -- 172.21.15.153:0/2915832522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80095f0 con 0x7f54f8100920 2024-05-06T14:23:28.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54f6d9d700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8194f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54e80092e0 tx=0x7f54e8009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:28.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54e801d070 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.651+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f54e800be10 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.652+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f8195690 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.652+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54e800f460 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.652+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f8195b30 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.652+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f54e800f6d0 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.652+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f818eb10 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.653+0000 7f54dffff700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f54e0038490 0x7f54e003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.653+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f54e804c490 con 0x7f54f8100920 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.653+0000 7f54f659c700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f54e0038490 0x7f54e003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.653+0000 7f54f659c700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f54e0038490 0x7f54e003a940 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006fd0 tx=0x7f54ec006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:28.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.657+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f54e8005c00 con 0x7f54f8100920 2024-05-06T14:23:28.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.855+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f54f8061220 con 0x7f54e0038490 2024-05-06T14:23:28.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.859+0000 7f54dffff700 1 -- 172.21.15.153:0/2915832522 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f54f8061220 con 0x7f54e0038490 2024-05-06T14:23:28.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f54e0038490 msgr2=0x7f54e003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:28.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f54e0038490 0x7f54e003a940 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006fd0 tx=0x7f54ec006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:28.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 msgr2=0x7f54f8194f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:28.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8194f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54e80092e0 tx=0x7f54e8009e60 comp rx=0 tx=0).stop 2024-05-06T14:23:28.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 shutdown_connections 2024-05-06T14:23:28.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f54e0038490 0x7f54e003a940 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:28.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 --2- 172.21.15.153:0/2915832522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f8100920 0x7f54f8194f50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:28.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 >> 172.21.15.153:0/2915832522 conn(0x7f54f80fa2e0 msgr2=0x7f54f80faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:28.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 shutdown_connections 2024-05-06T14:23:28.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:28.861+0000 7f54fd19b700 1 -- 172.21.15.153:0/2915832522 wait complete. 2024-05-06T14:23:29.451 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-05-06T14:23:29.451 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:23:29.451 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-05-06T14:23:29.470 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:23:29.470 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:23:29.528 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-05-06T14:23:29.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch host add smithi190 2024-05-06T14:23:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:23:30.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:30.150 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-05-06T14:23:30.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:29 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:23:31.099 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:30 smithi153 ceph-mon[27177]: Deploying daemon crash.smithi153 on smithi153 2024-05-06T14:23:31.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.398+0000 7fa44528c700 1 -- 172.21.15.153:0/1992180020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 msgr2=0x7fa4400fe500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:31.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.398+0000 7fa44528c700 1 --2- 172.21.15.153:0/1992180020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4400fe500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa430009910 tx=0x7fa430009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 -- 172.21.15.153:0/1992180020 shutdown_connections 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 --2- 172.21.15.153:0/1992180020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4400fe500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 -- 172.21.15.153:0/1992180020 >> 172.21.15.153:0/1992180020 conn(0x7fa4400f9a60 msgr2=0x7fa4400fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 -- 172.21.15.153:0/1992180020 shutdown_connections 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 -- 172.21.15.153:0/1992180020 wait complete. 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 Processor -- start 2024-05-06T14:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.399+0000 7fa44528c700 1 -- start start 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa44528c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4401971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa44528c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4401976e0 con 0x7fa4400fe0f0 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4401971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4401971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53684/0 (socket says 172.21.15.153:53684) 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43effd700 1 -- 172.21.15.153:0/3325438362 learned_addr learned my addr 172.21.15.153:0/3325438362 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43effd700 1 -- 172.21.15.153:0/3325438362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4300095f0 con 0x7fa4400fe0f0 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43effd700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4401971a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa430005b40 tx=0x7fa430004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:31.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa43001d070 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4401978e0 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa43000be10 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa43000f460 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.400+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa440197d80 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.401+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7fa43000f6d0 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.401+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa440190f10 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.401+0000 7fa43cff9700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fa42c038490 0x7fa42c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.401+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa43004c5a0 con 0x7fa4400fe0f0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.402+0000 7fa4367fc700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fa42c038490 0x7fa42c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.402+0000 7fa4367fc700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fa42c038490 0x7fa42c03a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa428006fd0 tx=0x7fa428006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:31.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.406+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa43002a360 con 0x7fa4400fe0f0 2024-05-06T14:23:31.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:31.604+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7fa440061190 con 0x7fa42c038490 2024-05-06T14:23:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:32 smithi153 ceph-mon[27177]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: Deploying cephadm binary to smithi190 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:34.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:33 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:34.049 INFO:teuthology.orchestra.run.smithi153.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-05-06T14:23:34.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.059+0000 7fa43cff9700 1 -- 172.21.15.153:0/3325438362 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa440061190 con 0x7fa42c038490 2024-05-06T14:23:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fa42c038490 msgr2=0x7fa42c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fa42c038490 0x7fa42c03a940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa428006fd0 tx=0x7fa428006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 msgr2=0x7fa4401971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4401971a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa430005b40 tx=0x7fa430004740 comp rx=0 tx=0).stop 2024-05-06T14:23:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 shutdown_connections 2024-05-06T14:23:34.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fa42c038490 0x7fa42c03a940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:34.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 --2- 172.21.15.153:0/3325438362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4400fe0f0 0x7fa4401971a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:34.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 >> 172.21.15.153:0/3325438362 conn(0x7fa4400f9a60 msgr2=0x7fa4400fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:34.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 shutdown_connections 2024-05-06T14:23:34.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:34.062+0000 7fa44528c700 1 -- 172.21.15.153:0/3325438362 wait complete. 2024-05-06T14:23:34.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-06T14:23:34.908 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:23:35.166 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:35 smithi153 ceph-mon[27177]: Deploying daemon node-exporter.smithi153 on smithi153 2024-05-06T14:23:35.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:35 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:35.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:35 smithi153 ceph-mon[27177]: Added host smithi190 2024-05-06T14:23:36.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:36 smithi153 ceph-mon[27177]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 -- 172.21.15.153:0/408972007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 msgr2=0x7f4684102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 --2- 172.21.15.153:0/408972007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684102d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4674009910 tx=0x7f4674009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 -- 172.21.15.153:0/408972007 shutdown_connections 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 --2- 172.21.15.153:0/408972007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684102d40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 -- 172.21.15.153:0/408972007 >> 172.21.15.153:0/408972007 conn(0x7f46840fa2c0 msgr2=0x7f46840fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 -- 172.21.15.153:0/408972007 shutdown_connections 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.319+0000 7f468c67b700 1 -- 172.21.15.153:0/408972007 wait complete. 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468c67b700 1 Processor -- start 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468c67b700 1 -- start start 2024-05-06T14:23:37.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468c67b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468c67b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841954b0 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468a417700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468a417700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53704/0 (socket says 172.21.15.153:53704) 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468a417700 1 -- 172.21.15.153:0/1206737993 learned_addr learned my addr 172.21.15.153:0/1206737993 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468a417700 1 -- 172.21.15.153:0/1206737993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740095f0 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.320+0000 7f468a417700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684194f70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4674005b40 tx=0x7f4674004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.321+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f467401d070 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.321+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f467400be10 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.321+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f467400f460 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.321+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46841956b0 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.321+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4684195b50 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.321+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f46740227e0 con 0x7f4684100960 2024-05-06T14:23:37.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.322+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468418eb10 con 0x7f4684100960 2024-05-06T14:23:37.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.322+0000 7f467b7fe700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4670038490 0x7f467003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:37.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.322+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f467404c4b0 con 0x7f4684100960 2024-05-06T14:23:37.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.322+0000 7f4689c16700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4670038490 0x7f467003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:37.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.322+0000 7f4689c16700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4670038490 0x7f467003a940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4680006fd0 tx=0x7f4680006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:37.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.326+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f467402bba0 con 0x7f4684100960 2024-05-06T14:23:37.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.523+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4684061190 con 0x7f4670038490 2024-05-06T14:23:37.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.525+0000 7f467b7fe700 1 -- 172.21.15.153:0/1206737993 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f4684061190 con 0x7f4670038490 2024-05-06T14:23:37.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:23:37.513 INFO:teuthology.orchestra.run.smithi153.stdout:[{"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4670038490 msgr2=0x7f467003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4670038490 0x7f467003a940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4680006fd0 tx=0x7f4680006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 msgr2=0x7f4684194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684194f70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4674005b40 tx=0x7f4674004740 comp rx=0 tx=0).stop 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 shutdown_connections 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4670038490 0x7f467003a940 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 --2- 172.21.15.153:0/1206737993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4684100960 0x7f4684194f70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:37.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 >> 172.21.15.153:0/1206737993 conn(0x7f46840fa2c0 msgr2=0x7f46840faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:37.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 shutdown_connections 2024-05-06T14:23:37.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:37.527+0000 7f468c67b700 1 -- 172.21.15.153:0/1206737993 wait complete. 2024-05-06T14:23:38.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:38 smithi153 ceph-mon[27177]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:38.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:38 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:38.577 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-06T14:23:38.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-06T14:23:38.797 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:23:39.218 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:39 smithi153 ceph-mon[27177]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T14:23:39.218 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:39 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:39.218 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:39 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:39.218 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:39 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:39.218 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:39 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:40.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:40 smithi153 ceph-mon[27177]: Deploying daemon alertmanager.smithi153 on smithi153 2024-05-06T14:23:40.274 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:40 smithi153 ceph-mon[27177]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:40.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 -- 172.21.15.153:0/3467155197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 msgr2=0x7fd8e0100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 --2- 172.21.15.153:0/3467155197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0100250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009910 tx=0x7fd8d0009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 -- 172.21.15.153:0/3467155197 shutdown_connections 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 --2- 172.21.15.153:0/3467155197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0100250 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 -- 172.21.15.153:0/3467155197 >> 172.21.15.153:0/3467155197 conn(0x7fd8e00fb3f0 msgr2=0x7fd8e00fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 -- 172.21.15.153:0/3467155197 shutdown_connections 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 -- 172.21.15.153:0/3467155197 wait complete. 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.546+0000 7fd8e64b4700 1 Processor -- start 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8e64b4700 1 -- start start 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8e64b4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8e64b4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e01954e0 con 0x7fd8e00ffe40 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8dffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8dffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0194fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41194/0 (socket says 172.21.15.153:41194) 2024-05-06T14:23:40.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8dffff700 1 -- 172.21.15.153:0/2414836200 learned_addr learned my addr 172.21.15.153:0/2414836200 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8dffff700 1 -- 172.21.15.153:0/2414836200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00095f0 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.547+0000 7fd8dffff700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0194fa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00092e0 tx=0x7fd8d0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.548+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d001d070 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.548+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e01956e0 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.548+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8d000be10 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.548+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d000f460 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.548+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e0195b80 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.549+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7fd8d000f6d0 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.549+0000 7fd8dd7fa700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fd8c8038490 0x7fd8c803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.549+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e018eaa0 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.549+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd8d004c310 con 0x7fd8e00ffe40 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.549+0000 7fd8df7fe700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fd8c8038490 0x7fd8c803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.549+0000 7fd8df7fe700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fd8c8038490 0x7fd8c803a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4006fd0 tx=0x7fd8d4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.554+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd8d0020960 con 0x7fd8e00ffe40 2024-05-06T14:23:40.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:40.759+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd8e002cc30 con 0x7fd8e00ffe40 2024-05-06T14:23:41.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.071+0000 7fd8dd7fa700 1 -- 172.21.15.153:0/2414836200 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd8d0027070 con 0x7fd8e00ffe40 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fd8c8038490 msgr2=0x7fd8c803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fd8c8038490 0x7fd8c803a940 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4006fd0 tx=0x7fd8d4006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 msgr2=0x7fd8e0194fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0194fa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00092e0 tx=0x7fd8d0009e60 comp rx=0 tx=0).stop 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 shutdown_connections 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fd8c8038490 0x7fd8c803a940 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 --2- 172.21.15.153:0/2414836200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00ffe40 0x7fd8e0194fa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 >> 172.21.15.153:0/2414836200 conn(0x7fd8e00fb3f0 msgr2=0x7fd8e00fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:41.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.073+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 shutdown_connections 2024-05-06T14:23:41.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:23:41.074+0000 7fd8e64b4700 1 -- 172.21.15.153:0/2414836200 wait complete. 2024-05-06T14:23:41.063 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-05-06T14:23:41.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:41 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2414836200' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-06T14:23:42.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:42 smithi153 ceph-mon[27177]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:42.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:42 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2414836200' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-06T14:23:42.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:42 smithi153 ceph-mon[27177]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T14:23:42.160 INFO:tasks.cephadm:Adding mon.smithi153 on smithi153 2024-05-06T14:23:42.160 INFO:tasks.cephadm:Adding mon.smithi190 on smithi190 2024-05-06T14:23:42.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch apply mon '2;smithi153:172.21.15.153=smithi153;smithi190:172.21.15.190=smithi190' 2024-05-06T14:23:42.431 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:42.478 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:43.940 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:43 smithi153 ceph-mon[27177]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:43.940 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:43 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:43.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/663021109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 msgr2=0x7fb0dc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:43.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 --2- 172.21.15.190:0/663021109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc102d20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800b0a0 tx=0x7fb0d800b3b0 comp rx=0 tx=0).stop 2024-05-06T14:23:43.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/663021109 shutdown_connections 2024-05-06T14:23:43.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 --2- 172.21.15.190:0/663021109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc102d20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:43.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/663021109 >> 172.21.15.190:0/663021109 conn(0x7fb0dc0fa2c0 msgr2=0x7fb0dc0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:43.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/663021109 shutdown_connections 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.969+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/663021109 wait complete. 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e2dd7700 1 Processor -- start 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e2dd7700 1 -- start start 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e2dd7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e2dd7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc1954b0 con 0x7fb0dc100940 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e0b73700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e0b73700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:41570/0 (socket says 172.21.15.190:41570) 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.970+0000 7fb0e0b73700 1 -- 172.21.15.190:0/235318553 learned_addr learned my addr 172.21.15.190:0/235318553 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:23:43.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0e0b73700 1 -- 172.21.15.190:0/235318553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8009d00 con 0x7fb0dc100940 2024-05-06T14:23:43.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0e0b73700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc194f70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8015040 tx=0x7fb0d800bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d800e040 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0dc1956b0 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0dc195ad0 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb0d801d930 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.971+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d8027980 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.972+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7fb0d8012430 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.972+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0dc18eb10 con 0x7fb0dc100940 2024-05-06T14:23:43.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.972+0000 7fb0d1ffb700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fb0c4038490 0x7fb0c403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:43.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.972+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb0d804cb80 con 0x7fb0dc100940 2024-05-06T14:23:43.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.973+0000 7fb0d3fff700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fb0c4038490 0x7fb0c403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:43.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.973+0000 7fb0d3fff700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fb0c4038490 0x7fb0c403a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc006fd0 tx=0x7fb0cc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:43.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:43.978+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb0d801b930 con 0x7fb0dc100940 2024-05-06T14:23:44.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.172+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 --> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0dc061240 con 0x7fb0c4038490 2024-05-06T14:23:44.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.179+0000 7fb0d1ffb700 1 -- 172.21.15.190:0/235318553 <== mgr.14164 v2:172.21.15.153:6800/2228103355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb0dc061240 con 0x7fb0c4038490 2024-05-06T14:23:44.168 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fb0c4038490 msgr2=0x7fb0c403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fb0c4038490 0x7fb0c403a940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc006fd0 tx=0x7fb0cc006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 msgr2=0x7fb0dc194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc194f70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8015040 tx=0x7fb0d800bb10 comp rx=0 tx=0).stop 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 shutdown_connections 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7fb0c4038490 0x7fb0c403a940 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 --2- 172.21.15.190:0/235318553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0dc100940 0x7fb0dc194f70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 >> 172.21.15.190:0/235318553 conn(0x7fb0dc0fa2c0 msgr2=0x7fb0dc0faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 shutdown_connections 2024-05-06T14:23:44.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:44.181+0000 7fb0e2dd7700 1 -- 172.21.15.190:0/235318553 wait complete. 2024-05-06T14:23:44.800 DEBUG:teuthology.orchestra.run.smithi190:mon.smithi190> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi190.service 2024-05-06T14:23:44.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:23:44.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:23:45.100 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:45.147 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:45.445 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:45 smithi153 ceph-mon[27177]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:23:45.446 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:45 smithi153 ceph-mon[27177]: Saving service mon spec with placement smithi153:172.21.15.153=smithi153;smithi190:172.21.15.190=smithi190;count:2 2024-05-06T14:23:45.446 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:45 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:45.446 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:45 smithi153 ceph-mon[27177]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:46.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 -- 172.21.15.190:0/4112963898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 msgr2=0x7f92340fe8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 --2- 172.21.15.190:0/4112963898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f92340fe8c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9224009910 tx=0x7f9224009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 -- 172.21.15.190:0/4112963898 shutdown_connections 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 --2- 172.21.15.190:0/4112963898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f92340fe8c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 -- 172.21.15.190:0/4112963898 >> 172.21.15.190:0/4112963898 conn(0x7f92340f9a60 msgr2=0x7f92340fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 -- 172.21.15.190:0/4112963898 shutdown_connections 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.579+0000 7f923a3e7700 1 -- 172.21.15.190:0/4112963898 wait complete. 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f923a3e7700 1 Processor -- start 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f923a3e7700 1 -- start start 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f923a3e7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f9234194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f923a3e7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234195500 con 0x7f92340fe4b0 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f9233fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f9234194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f9233fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f9234194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:41586/0 (socket says 172.21.15.190:41586) 2024-05-06T14:23:46.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.580+0000 7f9233fff700 1 -- 172.21.15.190:0/124583721 learned_addr learned my addr 172.21.15.190:0/124583721 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f9233fff700 1 -- 172.21.15.190:0/124583721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92240095f0 con 0x7f92340fe4b0 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f9233fff700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f9234194fc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9224005b40 tx=0x7f9224004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f922401d070 con 0x7f92340fe4b0 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f922400be10 con 0x7f92340fe4b0 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9234195700 con 0x7f92340fe4b0 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f922400f460 con 0x7f92340fe4b0 2024-05-06T14:23:46.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.581+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9234195ba0 con 0x7f92340fe4b0 2024-05-06T14:23:46.571 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.582+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f922400f6d0 con 0x7f92340fe4b0 2024-05-06T14:23:46.571 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.582+0000 7f92317fa700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f921c038490 0x7f921c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:46.571 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.582+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f922404c4b0 con 0x7f92340fe4b0 2024-05-06T14:23:46.571 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.582+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923418ea20 con 0x7f92340fe4b0 2024-05-06T14:23:46.571 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.582+0000 7f92337fe700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f921c038490 0x7f921c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:46.571 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.582+0000 7f92337fe700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f921c038490 0x7f921c03a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9228006fd0 tx=0x7f9228006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:46.574 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.587+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9224027070 con 0x7f92340fe4b0 2024-05-06T14:23:46.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.856+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f923402cc30 con 0x7f92340fe4b0 2024-05-06T14:23:46.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.856+0000 7f92317fa700 1 -- 172.21.15.190:0/124583721 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f922402c440 con 0x7f92340fe4b0 2024-05-06T14:23:46.844 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:23:46.844 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f921c038490 msgr2=0x7f921c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f921c038490 0x7f921c03a940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9228006fd0 tx=0x7f9228006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 msgr2=0x7f9234194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f9234194fc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9224005b40 tx=0x7f9224004740 comp rx=0 tx=0).stop 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 shutdown_connections 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f921c038490 0x7f921c03a940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 --2- 172.21.15.190:0/124583721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340fe4b0 0x7f9234194fc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:46.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 >> 172.21.15.190:0/124583721 conn(0x7f92340f9a60 msgr2=0x7f92341042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:46.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.858+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 shutdown_connections 2024-05-06T14:23:46.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:46.859+0000 7f923a3e7700 1 -- 172.21.15.190:0/124583721 wait complete. 2024-05-06T14:23:46.847 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/124583721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T14:23:48.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:47 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:23:48.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:23:48.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:23:48.626 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:48.687 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:49.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:48 smithi153 ceph-mon[27177]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T14:23:49.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:48 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T14:23:49.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:48 smithi153 ceph-mon[27177]: Deploying daemon grafana.smithi153 on smithi153 2024-05-06T14:23:50.141 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 -- 172.21.15.190:0/3279815456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 msgr2=0x7f15600fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:50.141 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 --2- 172.21.15.190:0/3279815456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f15600fe4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f155c009910 tx=0x7f155c009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:50.141 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 -- 172.21.15.190:0/3279815456 shutdown_connections 2024-05-06T14:23:50.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 --2- 172.21.15.190:0/3279815456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f15600fe4f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:50.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 -- 172.21.15.190:0/3279815456 >> 172.21.15.190:0/3279815456 conn(0x7f15600f9af0 msgr2=0x7f15600fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:50.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 -- 172.21.15.190:0/3279815456 shutdown_connections 2024-05-06T14:23:50.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.149+0000 7f1567a79700 1 -- 172.21.15.190:0/3279815456 wait complete. 2024-05-06T14:23:50.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1567a79700 1 Processor -- start 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1567a79700 1 -- start start 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1567a79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f156019b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1567a79700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156019bae0 con 0x7f15600fe0e0 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1565815700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f156019b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1565815700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f156019b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:59414/0 (socket says 172.21.15.190:59414) 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1565815700 1 -- 172.21.15.190:0/127713616 learned_addr learned my addr 172.21.15.190:0/127713616 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:23:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1565815700 1 -- 172.21.15.190:0/127713616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0095f0 con 0x7f15600fe0e0 2024-05-06T14:23:50.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.150+0000 7f1565815700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f156019b5a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f155c0092e0 tx=0x7f155c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:50.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f155c01d070 con 0x7f15600fe0e0 2024-05-06T14:23:50.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f155c00be10 con 0x7f15600fe0e0 2024-05-06T14:23:50.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f155c00f460 con 0x7f15600fe0e0 2024-05-06T14:23:50.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156019bce0 con 0x7f15600fe0e0 2024-05-06T14:23:50.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156019c180 con 0x7f15600fe0e0 2024-05-06T14:23:50.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f155c022470 con 0x7f15600fe0e0 2024-05-06T14:23:50.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.151+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1560195240 con 0x7f15600fe0e0 2024-05-06T14:23:50.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.152+0000 7f1556ffd700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f154c0384e0 0x7f154c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:50.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.152+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f155c04c510 con 0x7f15600fe0e0 2024-05-06T14:23:50.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.152+0000 7f1565014700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f154c0384e0 0x7f154c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:50.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.152+0000 7f1565014700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f154c0384e0 0x7f154c03a990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1550006fd0 tx=0x7f1550006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:50.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.157+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f155c00b960 con 0x7f15600fe0e0 2024-05-06T14:23:50.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:49 smithi153 ceph-mon[27177]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:50.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.419+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f156002cc30 con 0x7f15600fe0e0 2024-05-06T14:23:50.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.420+0000 7f1556ffd700 1 -- 172.21.15.190:0/127713616 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f155c027030 con 0x7f15600fe0e0 2024-05-06T14:23:50.408 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:23:50.408 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f154c0384e0 msgr2=0x7f154c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f154c0384e0 0x7f154c03a990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1550006fd0 tx=0x7f1550006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 msgr2=0x7f156019b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f156019b5a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f155c0092e0 tx=0x7f155c009e60 comp rx=0 tx=0).stop 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 shutdown_connections 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f154c0384e0 0x7f154c03a990 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 --2- 172.21.15.190:0/127713616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15600fe0e0 0x7f156019b5a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 >> 172.21.15.190:0/127713616 conn(0x7f15600f9af0 msgr2=0x7f15601017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 shutdown_connections 2024-05-06T14:23:50.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:50.422+0000 7f1567a79700 1 -- 172.21.15.190:0/127713616 wait complete. 2024-05-06T14:23:50.411 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:23:51.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:50 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/127713616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:23:51.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:23:51.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:23:52.191 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:52.238 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:52.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:51 smithi153 ceph-mon[27177]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:53.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.772+0000 7efc79a3c700 1 -- 172.21.15.190:0/2344414413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 msgr2=0x7efc740ff0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:53.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.772+0000 7efc79a3c700 1 --2- 172.21.15.190:0/2344414413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc740ff0d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efc64009910 tx=0x7efc64009c20 comp rx=0 tx=0).stop 2024-05-06T14:23:53.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 -- 172.21.15.190:0/2344414413 shutdown_connections 2024-05-06T14:23:53.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 --2- 172.21.15.190:0/2344414413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc740ff0d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 -- 172.21.15.190:0/2344414413 >> 172.21.15.190:0/2344414413 conn(0x7efc740fa250 msgr2=0x7efc740fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 -- 172.21.15.190:0/2344414413 shutdown_connections 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 -- 172.21.15.190:0/2344414413 wait complete. 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 Processor -- start 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.773+0000 7efc79a3c700 1 -- start start 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc79a3c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc74193e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc79a3c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74194390 con 0x7efc740fecc0 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc72ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc74193e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc72ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc74193e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:59428/0 (socket says 172.21.15.190:59428) 2024-05-06T14:23:53.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc72ffd700 1 -- 172.21.15.190:0/1512990323 learned_addr learned my addr 172.21.15.190:0/1512990323 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc72ffd700 1 -- 172.21.15.190:0/1512990323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc640095f0 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc72ffd700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc74193e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc64005b40 tx=0x7efc64004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6401d070 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efc6400be10 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc74194590 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6400f460 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.774+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc74194a30 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.775+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7efc6400f6d0 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.775+0000 7efc78a3a700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7efc5c0384f0 0x7efc5c03a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.775+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7418d9a0 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.775+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efc6404c4c0 con 0x7efc740fecc0 2024-05-06T14:23:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.776+0000 7efc727fc700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7efc5c0384f0 0x7efc5c03a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:53.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.776+0000 7efc727fc700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7efc5c0384f0 0x7efc5c03a9a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc68006fd0 tx=0x7efc68006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:53.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:53.780+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efc64027070 con 0x7efc740fecc0 2024-05-06T14:23:54.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.049+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc7402cc30 con 0x7efc740fecc0 2024-05-06T14:23:54.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.050+0000 7efc78a3a700 1 -- 172.21.15.190:0/1512990323 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7efc6402c440 con 0x7efc740fecc0 2024-05-06T14:23:54.038 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:23:54.038 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:23:54.039 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7efc5c0384f0 msgr2=0x7efc5c03a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:54.039 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7efc5c0384f0 0x7efc5c03a9a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc68006fd0 tx=0x7efc68006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:54.039 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 msgr2=0x7efc74193e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc74193e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc64005b40 tx=0x7efc64004740 comp rx=0 tx=0).stop 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 shutdown_connections 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7efc5c0384f0 0x7efc5c03a9a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 --2- 172.21.15.190:0/1512990323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc740fecc0 0x7efc74193e50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 >> 172.21.15.190:0/1512990323 conn(0x7efc740fa250 msgr2=0x7efc740faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 shutdown_connections 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:54.052+0000 7efc79a3c700 1 -- 172.21.15.190:0/1512990323 wait complete. 2024-05-06T14:23:54.040 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:23:54.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:53 smithi153 ceph-mon[27177]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:55.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:54 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1512990323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:23:55.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:23:55.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:23:55.806 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:55.854 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:56.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:55 smithi153 ceph-mon[27177]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:57.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 -- 172.21.15.190:0/1049966977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 msgr2=0x7faaf80ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 --2- 172.21.15.190:0/1049966977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf80ff140 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b0a0 tx=0x7faaf000b3b0 comp rx=0 tx=0).stop 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 -- 172.21.15.190:0/1049966977 shutdown_connections 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 --2- 172.21.15.190:0/1049966977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf80ff140 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 -- 172.21.15.190:0/1049966977 >> 172.21.15.190:0/1049966977 conn(0x7faaf80fa2c0 msgr2=0x7faaf80fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 -- 172.21.15.190:0/1049966977 shutdown_connections 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.199+0000 7faaff26b700 1 -- 172.21.15.190:0/1049966977 wait complete. 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faaff26b700 1 Processor -- start 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faaff26b700 1 -- start start 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faaff26b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faaff26b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8197d40 con 0x7faaf80fed30 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faafd007700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faafd007700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf8069350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:59446/0 (socket says 172.21.15.190:59446) 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faafd007700 1 -- 172.21.15.190:0/3042121275 learned_addr learned my addr 172.21.15.190:0/3042121275 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.200+0000 7faafd007700 1 -- 172.21.15.190:0/3042121275 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0009d00 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faafd007700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf8069350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b830 tx=0x7faaf000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaf000e040 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faaf0012af0 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf8197e80 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaf001d9f0 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8198240 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.201+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7faaf001dc60 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.202+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf818da10 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.202+0000 7faaf67fc700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faae00384e0 0x7faae003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.202+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faaf004c880 con 0x7faaf80fed30 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.202+0000 7faafc806700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faae00384e0 0x7faae003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:23:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.202+0000 7faafc806700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faae00384e0 0x7faae003a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faaec006fd0 tx=0x7faaec006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:23:57.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.207+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7faaf001c3e0 con 0x7faaf80fed30 2024-05-06T14:23:57.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.476+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faaf802cc30 con 0x7faaf80fed30 2024-05-06T14:23:57.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.476+0000 7faaf67fc700 1 -- 172.21.15.190:0/3042121275 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7faaf001c3e0 con 0x7faaf80fed30 2024-05-06T14:23:57.464 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:23:57.464 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:23:57.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faae00384e0 msgr2=0x7faae003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:57.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faae00384e0 0x7faae003a990 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faaec006fd0 tx=0x7faaec006e40 comp rx=0 tx=0).stop 2024-05-06T14:23:57.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 msgr2=0x7faaf8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:23:57.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf8069350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faaf000b830 tx=0x7faaf000bd40 comp rx=0 tx=0).stop 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 shutdown_connections 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7faae00384e0 0x7faae003a990 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 --2- 172.21.15.190:0/3042121275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf80fed30 0x7faaf8069350 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 >> 172.21.15.190:0/3042121275 conn(0x7faaf80fa2c0 msgr2=0x7faaf80faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.478+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 shutdown_connections 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:23:57.479+0000 7faaff26b700 1 -- 172.21.15.190:0/3042121275 wait complete. 2024-05-06T14:23:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:23:58.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:57 smithi153 ceph-mon[27177]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:23:58.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:57 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/3042121275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:23:59.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:23:59.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:23:59.357 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:23:59.405 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:00.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:23:59 smithi153 ceph-mon[27177]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:00.885 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 -- 172.21.15.190:0/2394474006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 msgr2=0x7f66dc0ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 --2- 172.21.15.190:0/2394474006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc0ff140 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009910 tx=0x7f66cc009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 -- 172.21.15.190:0/2394474006 shutdown_connections 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 --2- 172.21.15.190:0/2394474006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc0ff140 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 -- 172.21.15.190:0/2394474006 >> 172.21.15.190:0/2394474006 conn(0x7f66dc0fa2e0 msgr2=0x7f66dc0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 -- 172.21.15.190:0/2394474006 shutdown_connections 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.896+0000 7f66e3c72700 1 -- 172.21.15.190:0/2394474006 wait complete. 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e3c72700 1 Processor -- start 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e3c72700 1 -- start start 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e3c72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc193ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:00.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e3c72700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc194410 con 0x7f66dc0fed30 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e1a0e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc193ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e1a0e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc193ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:51898/0 (socket says 172.21.15.190:51898) 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e1a0e700 1 -- 172.21.15.190:0/2700739857 learned_addr learned my addr 172.21.15.190:0/2700739857 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.897+0000 7f66e1a0e700 1 -- 172.21.15.190:0/2700739857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc0095f0 con 0x7f66dc0fed30 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.898+0000 7f66e1a0e700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc193ed0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f66cc005b40 tx=0x7f66cc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.898+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66cc01d070 con 0x7f66dc0fed30 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.898+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f66cc00be10 con 0x7f66dc0fed30 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.898+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc194610 con 0x7f66dc0fed30 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.898+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66cc00f460 con 0x7f66dc0fed30 2024-05-06T14:24:00.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.898+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc194ab0 con 0x7f66dc0fed30 2024-05-06T14:24:00.888 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.899+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc18da10 con 0x7f66dc0fed30 2024-05-06T14:24:00.888 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.899+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f66cc0227e0 con 0x7f66dc0fed30 2024-05-06T14:24:00.888 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.899+0000 7f66d2ffd700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f66c8038490 0x7f66c803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:00.888 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.899+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f66cc04c4a0 con 0x7f66dc0fed30 2024-05-06T14:24:00.888 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.899+0000 7f66e120d700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f66c8038490 0x7f66c803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:00.889 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.899+0000 7f66e120d700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f66c8038490 0x7f66c803a940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66d8006fd0 tx=0x7f66d8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:00.891 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:00.904+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f66cc02bba0 con 0x7f66dc0fed30 2024-05-06T14:24:01.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.169+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66dc02cc30 con 0x7f66dc0fed30 2024-05-06T14:24:01.157 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.169+0000 7f66d2ffd700 1 -- 172.21.15.190:0/2700739857 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f66cc027030 con 0x7f66dc0fed30 2024-05-06T14:24:01.157 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:01.157 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f66c8038490 msgr2=0x7f66c803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f66c8038490 0x7f66c803a940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66d8006fd0 tx=0x7f66d8006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 msgr2=0x7f66dc193ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc193ed0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f66cc005b40 tx=0x7f66cc004740 comp rx=0 tx=0).stop 2024-05-06T14:24:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 shutdown_connections 2024-05-06T14:24:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f66c8038490 0x7f66c803a940 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:01.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 --2- 172.21.15.190:0/2700739857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66dc0fed30 0x7f66dc193ed0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:01.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 >> 172.21.15.190:0/2700739857 conn(0x7f66dc0fa2e0 msgr2=0x7f66dc0faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:01.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 shutdown_connections 2024-05-06T14:24:01.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:01.172+0000 7f66e3c72700 1 -- 172.21.15.190:0/2700739857 wait complete. 2024-05-06T14:24:01.160 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:02.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:01 smithi153 ceph-mon[27177]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:02.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:01 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/2700739857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:02.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:02.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:02.939 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:02.987 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:03 smithi153 ceph-mon[27177]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/1402584586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 msgr2=0x7f7bc40fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 --2- 172.21.15.190:0/1402584586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc40fe4f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009910 tx=0x7f7bb4009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/1402584586 shutdown_connections 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 --2- 172.21.15.190:0/1402584586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc40fe4f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/1402584586 >> 172.21.15.190:0/1402584586 conn(0x7f7bc40f9af0 msgr2=0x7f7bc40fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/1402584586 shutdown_connections 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.499+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/1402584586 wait complete. 2024-05-06T14:24:04.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc8d4d700 1 Processor -- start 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc8d4d700 1 -- start start 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc8d4d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc4194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc8d4d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc41954b0 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc4194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc4194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:51904/0 (socket says 172.21.15.190:51904) 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.500+0000 7f7bc259c700 1 -- 172.21.15.190:0/3171157659 learned_addr learned my addr 172.21.15.190:0/3171157659 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:04.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bc259c700 1 -- 172.21.15.190:0/3171157659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40095f0 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bc259c700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc4194f70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005b40 tx=0x7f7bb4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:04.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb401d070 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc41956b0 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7bb400be10 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bb400f460 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.501+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc4195b50 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.502+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f7bb400b960 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.502+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc418ea80 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.502+0000 7f7bb37fe700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f7bac038490 0x7f7bac03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.502+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7bb404c4c0 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.502+0000 7f7bc1d9b700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f7bac038490 0x7f7bac03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:04.492 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.503+0000 7f7bc1d9b700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f7bac038490 0x7f7bac03a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8006fd0 tx=0x7f7bb8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:04.495 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.507+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7bb402a360 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.774+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bc402cc30 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.763 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.775+0000 7f7bb37fe700 1 -- 172.21.15.190:0/3171157659 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7bb4027030 con 0x7f7bc40fe0e0 2024-05-06T14:24:04.763 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:04.763 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.776+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f7bac038490 msgr2=0x7f7bac03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.776+0000 7f7bc8d4d700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f7bac038490 0x7f7bac03a940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8006fd0 tx=0x7f7bb8006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.776+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 msgr2=0x7f7bc4194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.776+0000 7f7bc8d4d700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc4194f70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005b40 tx=0x7f7bb4004740 comp rx=0 tx=0).stop 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.777+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 shutdown_connections 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.777+0000 7f7bc8d4d700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f7bac038490 0x7f7bac03a940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:04.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.777+0000 7f7bc8d4d700 1 --2- 172.21.15.190:0/3171157659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bc40fe0e0 0x7f7bc4194f70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:04.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.777+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 >> 172.21.15.190:0/3171157659 conn(0x7f7bc40f9af0 msgr2=0x7f7bc40fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:04.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.777+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 shutdown_connections 2024-05-06T14:24:04.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:04.777+0000 7f7bc8d4d700 1 -- 172.21.15.190:0/3171157659 wait complete. 2024-05-06T14:24:04.766 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:05.155 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:04 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/3171157659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:06.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:05 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:06.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:06.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:06.525 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:06.572 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:07.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:06 smithi153 ceph-mon[27177]: Deploying daemon prometheus.smithi153 on smithi153 2024-05-06T14:24:08.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:07 smithi153 ceph-mon[27177]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:08.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:07 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.082+0000 7f57424d6700 1 -- 172.21.15.190:0/1130281154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 msgr2=0x7f573c100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.082+0000 7f57424d6700 1 --2- 172.21.15.190:0/1130281154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c100250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f572c009910 tx=0x7f572c009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.082+0000 7f57424d6700 1 -- 172.21.15.190:0/1130281154 shutdown_connections 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.082+0000 7f57424d6700 1 --2- 172.21.15.190:0/1130281154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c100250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.082+0000 7f57424d6700 1 -- 172.21.15.190:0/1130281154 >> 172.21.15.190:0/1130281154 conn(0x7f573c0fb3d0 msgr2=0x7f573c0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.082+0000 7f57424d6700 1 -- 172.21.15.190:0/1130281154 shutdown_connections 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.083+0000 7f57424d6700 1 -- 172.21.15.190:0/1130281154 wait complete. 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.083+0000 7f57424d6700 1 Processor -- start 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.083+0000 7f57424d6700 1 -- start start 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.083+0000 7f57424d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.083+0000 7f57424d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c075030 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.084+0000 7f573bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.084+0000 7f573bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:51934/0 (socket says 172.21.15.190:51934) 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.084+0000 7f573bfff700 1 -- 172.21.15.190:0/2111171309 learned_addr learned my addr 172.21.15.190:0/2111171309 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.084+0000 7f573bfff700 1 -- 172.21.15.190:0/2111171309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0095f0 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.084+0000 7f573bfff700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c074af0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f572c005b40 tx=0x7f572c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.084+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c01d070 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.085+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f572c00be10 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.085+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c073140 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.085+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c00f460 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.085+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c0735e0 con 0x7f573c0ffe40 2024-05-06T14:24:08.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.085+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f572c00b960 con 0x7f573c0ffe40 2024-05-06T14:24:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.085+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c062380 con 0x7f573c0ffe40 2024-05-06T14:24:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.086+0000 7f57397fa700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f5724038490 0x7f572403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.086+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f572c04c360 con 0x7f573c0ffe40 2024-05-06T14:24:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.086+0000 7f573b7fe700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f5724038490 0x7f572403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.086+0000 7f573b7fe700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f5724038490 0x7f572403a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5730006fd0 tx=0x7f5730006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:08.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.090+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f572c02bba0 con 0x7f573c0ffe40 2024-05-06T14:24:08.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.355+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f573c18e480 con 0x7f573c0ffe40 2024-05-06T14:24:08.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.355+0000 7f57397fa700 1 -- 172.21.15.190:0/2111171309 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f572c027070 con 0x7f573c0ffe40 2024-05-06T14:24:08.344 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:08.345 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f5724038490 msgr2=0x7f572403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f5724038490 0x7f572403a940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5730006fd0 tx=0x7f5730006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 msgr2=0x7f573c074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c074af0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f572c005b40 tx=0x7f572c004740 comp rx=0 tx=0).stop 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 shutdown_connections 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f5724038490 0x7f572403a940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 --2- 172.21.15.190:0/2111171309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f573c0ffe40 0x7f573c074af0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 >> 172.21.15.190:0/2111171309 conn(0x7f573c0fb3d0 msgr2=0x7f573c0fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 shutdown_connections 2024-05-06T14:24:08.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:08.357+0000 7f57424d6700 1 -- 172.21.15.190:0/2111171309 wait complete. 2024-05-06T14:24:08.347 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:09.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:08 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/2111171309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:09.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:09.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:10.115 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:10.164 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:10.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:09 smithi153 ceph-mon[27177]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 -- 172.21.15.190:0/1533276816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 msgr2=0x7f3d78102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 --2- 172.21.15.190:0/1533276816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78102ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009910 tx=0x7f3d68009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 -- 172.21.15.190:0/1533276816 shutdown_connections 2024-05-06T14:24:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 --2- 172.21.15.190:0/1533276816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78102ca0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 -- 172.21.15.190:0/1533276816 >> 172.21.15.190:0/1533276816 conn(0x7f3d780fa2e0 msgr2=0x7f3d780fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 -- 172.21.15.190:0/1533276816 shutdown_connections 2024-05-06T14:24:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.665+0000 7f3d7d6df700 1 -- 172.21.15.190:0/1533276816 wait complete. 2024-05-06T14:24:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d7d6df700 1 Processor -- start 2024-05-06T14:24:11.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d7d6df700 1 -- start start 2024-05-06T14:24:11.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d7d6df700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:11.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d7d6df700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781976a0 con 0x7f3d781008c0 2024-05-06T14:24:11.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:11.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:57486/0 (socket says 172.21.15.190:57486) 2024-05-06T14:24:11.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.666+0000 7f3d76ffd700 1 -- 172.21.15.190:0/2406844974 learned_addr learned my addr 172.21.15.190:0/2406844974 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:11.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d76ffd700 1 -- 172.21.15.190:0/2406844974 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d680095f0 con 0x7f3d781008c0 2024-05-06T14:24:11.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d76ffd700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78197160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3d68005b40 tx=0x7f3d68004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:11.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6801d070 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3d6800be10 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d781978a0 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6800f460 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.667+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d78197d40 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.668+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f3d6800f6d0 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.668+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d78190f10 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.668+0000 7f3d74ff9700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3d64038490 0x7f3d6403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.668+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d6804c5c0 con 0x7f3d781008c0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.668+0000 7f3d6e7fc700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3d64038490 0x7f3d6403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.669+0000 7f3d6e7fc700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3d64038490 0x7f3d6403a940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d60006fd0 tx=0x7f3d60006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:11.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.675+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3d6802be20 con 0x7f3d781008c0 2024-05-06T14:24:11.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.939+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d7802cc30 con 0x7f3d781008c0 2024-05-06T14:24:11.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.940+0000 7f3d74ff9700 1 -- 172.21.15.190:0/2406844974 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3d68027030 con 0x7f3d781008c0 2024-05-06T14:24:11.929 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:11.929 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:11.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3d64038490 msgr2=0x7f3d6403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3d64038490 0x7f3d6403a940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d60006fd0 tx=0x7f3d60006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 msgr2=0x7f3d78197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78197160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3d68005b40 tx=0x7f3d68004740 comp rx=0 tx=0).stop 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 shutdown_connections 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f3d64038490 0x7f3d6403a940 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 --2- 172.21.15.190:0/2406844974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d781008c0 0x7f3d78197160 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 >> 172.21.15.190:0/2406844974 conn(0x7f3d780fa2e0 msgr2=0x7f3d780fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 shutdown_connections 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:11.942+0000 7f3d7d6df700 1 -- 172.21.15.190:0/2406844974 wait complete. 2024-05-06T14:24:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:12.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:11 smithi153 ceph-mon[27177]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:12.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:11 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/2406844974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:13.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:13.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:13.645 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:13.692 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:14.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:13 smithi153 ceph-mon[27177]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- 172.21.15.190:0/1622195314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 msgr2=0x7f2ef80fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 --2- 172.21.15.190:0/1622195314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef80fe4f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009910 tx=0x7f2eec009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- 172.21.15.190:0/1622195314 shutdown_connections 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 --2- 172.21.15.190:0/1622195314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef80fe4f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- 172.21.15.190:0/1622195314 >> 172.21.15.190:0/1622195314 conn(0x7f2ef80f9af0 msgr2=0x7f2ef80fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- 172.21.15.190:0/1622195314 shutdown_connections 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- 172.21.15.190:0/1622195314 wait complete. 2024-05-06T14:24:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 Processor -- start 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- start start 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef8194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.139+0000 7f2efeea7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef81954b0 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2efcc43700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef8194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2efcc43700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef8194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:57506/0 (socket says 172.21.15.190:57506) 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2efcc43700 1 -- 172.21.15.190:0/3359015038 learned_addr learned my addr 172.21.15.190:0/3359015038 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2efcc43700 1 -- 172.21.15.190:0/3359015038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec0095f0 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2efcc43700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef8194f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2eec005b40 tx=0x7f2eec004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eec01d070 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef81956b0 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2eec00be10 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.140+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eec00f460 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.141+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef8195b50 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.141+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef818ea80 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.141+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 45141+0+0 (secure 0 0 0) 0x7f2eec0227e0 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.142+0000 7f2ef5ffb700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f2ee8038490 0x7f2ee803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.142+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2eec04c4b0 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.142+0000 7f2ef7fff700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f2ee8038490 0x7f2ee803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.142+0000 7f2ef7fff700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f2ee8038490 0x7f2ee803a940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4006fd0 tx=0x7f2ee4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:15.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.147+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2eec02bba0 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.412+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ef802cc30 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.413+0000 7f2ef5ffb700 1 -- 172.21.15.190:0/3359015038 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2eec027030 con 0x7f2ef80fe0e0 2024-05-06T14:24:15.401 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:15.402 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f2ee8038490 msgr2=0x7f2ee803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f2ee8038490 0x7f2ee803a940 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4006fd0 tx=0x7f2ee4006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 msgr2=0x7f2ef8194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef8194f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2eec005b40 tx=0x7f2eec004740 comp rx=0 tx=0).stop 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 shutdown_connections 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f2ee8038490 0x7f2ee803a940 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 --2- 172.21.15.190:0/3359015038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ef80fe0e0 0x7f2ef8194f70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 >> 172.21.15.190:0/3359015038 conn(0x7f2ef80f9af0 msgr2=0x7f2ef80fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 shutdown_connections 2024-05-06T14:24:15.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:15.415+0000 7f2efeea7700 1 -- 172.21.15.190:0/3359015038 wait complete. 2024-05-06T14:24:15.406 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:16.278 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:15 smithi153 ceph-mon[27177]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:16.278 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:15 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/3359015038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:17.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:17.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:17.281 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:17.329 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:17 smithi153 ceph-mon[27177]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:17 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:17 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:17 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:17 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T14:24:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:17 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:18.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 -- 172.21.15.190:0/647760618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 msgr2=0x7f4350100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:18.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 --2- 172.21.15.190:0/647760618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350100280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4340009910 tx=0x7f4340009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:18.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 -- 172.21.15.190:0/647760618 shutdown_connections 2024-05-06T14:24:18.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 --2- 172.21.15.190:0/647760618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350100280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:18.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 -- 172.21.15.190:0/647760618 >> 172.21.15.190:0/647760618 conn(0x7f43500fb470 msgr2=0x7f43500fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:18.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 -- 172.21.15.190:0/647760618 shutdown_connections 2024-05-06T14:24:18.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.827+0000 7f4355655700 1 -- 172.21.15.190:0/647760618 wait complete. 2024-05-06T14:24:18.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.828+0000 7f4355655700 1 Processor -- start 2024-05-06T14:24:18.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.828+0000 7f4355655700 1 -- start start 2024-05-06T14:24:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.828+0000 7f4355655700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.828+0000 7f4355655700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4350195500 con 0x7f43500ffe70 2024-05-06T14:24:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.828+0000 7f434effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.828+0000 7f434effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:57532/0 (socket says 172.21.15.190:57532) 2024-05-06T14:24:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f434effd700 1 -- 172.21.15.190:0/1068412053 learned_addr learned my addr 172.21.15.190:0/1068412053 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f434effd700 1 -- 172.21.15.190:0/1068412053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400095f0 con 0x7f43500ffe70 2024-05-06T14:24:18.820 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f434effd700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350194fc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4340005b40 tx=0x7f4340004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:18.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f434001d070 con 0x7f43500ffe70 2024-05-06T14:24:18.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f434000be10 con 0x7f43500ffe70 2024-05-06T14:24:18.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4350195700 con 0x7f43500ffe70 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f434000f460 con 0x7f43500ffe70 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.829+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4350195ba0 con 0x7f43500ffe70 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.830+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 14) v1 ==== 45155+0+0 (secure 0 0 0) 0x7f434000f5c0 con 0x7f43500ffe70 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.830+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435018ea70 con 0x7f43500ffe70 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.830+0000 7f4337fff700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 0x7f433803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.830+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f434004c590 con 0x7f43500ffe70 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.830+0000 7f434e7fc700 1 -- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 msgr2=0x7f433803a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2228103355 2024-05-06T14:24:18.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.830+0000 7f434e7fc700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 0x7f433803a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:24:18.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:18.835+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4340020550 con 0x7f43500ffe70 2024-05-06T14:24:19.019 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.031+0000 7f434e7fc700 1 -- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 msgr2=0x7f433803a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2228103355 2024-05-06T14:24:19.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.031+0000 7f434e7fc700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 0x7f433803a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-06T14:24:19.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.104+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f435002cc30 con 0x7f43500ffe70 2024-05-06T14:24:19.094 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.105+0000 7f4337fff700 1 -- 172.21.15.190:0/1068412053 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4340027020 con 0x7f43500ffe70 2024-05-06T14:24:19.094 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:19.094 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:19.096 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 msgr2=0x7f433803a9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:24:19.096 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 0x7f433803a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 msgr2=0x7f4350194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350194fc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4340005b40 tx=0x7f4340004740 comp rx=0 tx=0).stop 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 shutdown_connections 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f4338038530 0x7f433803a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 --2- 172.21.15.190:0/1068412053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43500ffe70 0x7f4350194fc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 >> 172.21.15.190:0/1068412053 conn(0x7f43500fb470 msgr2=0x7f43500fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 shutdown_connections 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:19.107+0000 7f4355655700 1 -- 172.21.15.190:0/1068412053 wait complete. 2024-05-06T14:24:19.097 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:19.631 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:19 smithi153 ceph-mon[27177]: from='mgr.14164 172.21.15.153:0/3219776986' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T14:24:19.631 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:19 smithi153 ceph-mon[27177]: mgrmap e14: smithi153.dolfed(active, since 65s) 2024-05-06T14:24:19.631 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:19 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1068412053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:20.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:20.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:20.861 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:20.908 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.196+0000 7f3849d39700 1 -- 172.21.15.190:0/4116882546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 msgr2=0x7f3844100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.196+0000 7f3849d39700 1 --2- 172.21.15.190:0/4116882546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844100290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3834009910 tx=0x7f3834009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.196+0000 7f3849d39700 1 -- 172.21.15.190:0/4116882546 shutdown_connections 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.196+0000 7f3849d39700 1 --2- 172.21.15.190:0/4116882546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844100290 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.196+0000 7f3849d39700 1 -- 172.21.15.190:0/4116882546 >> 172.21.15.190:0/4116882546 conn(0x7f38440fb4a0 msgr2=0x7f38440fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.197+0000 7f3849d39700 1 -- 172.21.15.190:0/4116882546 shutdown_connections 2024-05-06T14:24:22.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.197+0000 7f3849d39700 1 -- 172.21.15.190:0/4116882546 wait complete. 2024-05-06T14:24:22.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.197+0000 7f3849d39700 1 Processor -- start 2024-05-06T14:24:22.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.197+0000 7f3849d39700 1 -- start start 2024-05-06T14:24:22.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.197+0000 7f3849d39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844194f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:22.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.197+0000 7f3849d39700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441954d0 con 0x7f38440ffe80 2024-05-06T14:24:22.188 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f38437fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844194f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f38437fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844194f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:50952/0 (socket says 172.21.15.190:50952) 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f38437fe700 1 -- 172.21.15.190:0/3846764085 learned_addr learned my addr 172.21.15.190:0/3846764085 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f38437fe700 1 -- 172.21.15.190:0/3846764085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340095f0 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f38437fe700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844194f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3834005950 tx=0x7f3834009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f383401d070 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f383400be10 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38441956d0 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f383400f460 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.198+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3844195b70 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.199+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 14) v1 ==== 45155+0+0 (secure 0 0 0) 0x7f383400f5c0 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.199+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384418ea80 con 0x7f38440ffe80 2024-05-06T14:24:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.199+0000 7f3840ff9700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 0x7f382c03a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:22.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.200+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f383404c5f0 con 0x7f38440ffe80 2024-05-06T14:24:22.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.200+0000 7f3842ffd700 1 -- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 msgr2=0x7f382c03a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2228103355 2024-05-06T14:24:22.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.200+0000 7f3842ffd700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 0x7f382c03a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:24:22.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.204+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3834020550 con 0x7f38440ffe80 2024-05-06T14:24:22.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.401+0000 7f3842ffd700 1 -- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 msgr2=0x7f382c03a9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2228103355 2024-05-06T14:24:22.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.401+0000 7f3842ffd700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 0x7f382c03a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-06T14:24:22.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.469+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f384402cc30 con 0x7f38440ffe80 2024-05-06T14:24:22.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.469+0000 7f3840ff9700 1 -- 172.21.15.190:0/3846764085 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3834027030 con 0x7f38440ffe80 2024-05-06T14:24:22.457 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:22.457 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 msgr2=0x7f382c03a9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 0x7f382c03a9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 msgr2=0x7f3844194f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844194f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3834005950 tx=0x7f3834009e60 comp rx=0 tx=0).stop 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 shutdown_connections 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:6800/2228103355,v1:172.21.15.153:6801/2228103355] conn(0x7f382c038530 0x7f382c03a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:22.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 --2- 172.21.15.190:0/3846764085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38440ffe80 0x7f3844194f90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:22.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 >> 172.21.15.190:0/3846764085 conn(0x7f38440fb4a0 msgr2=0x7f38440fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:22.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 shutdown_connections 2024-05-06T14:24:22.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:22.471+0000 7f3849d39700 1 -- 172.21.15.190:0/3846764085 wait complete. 2024-05-06T14:24:22.460 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:22.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:22 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/3846764085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:23.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:23.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:24.155 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:24.206 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: Active manager daemon smithi153.dolfed restarted 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: Activating manager daemon smithi153.dolfed 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: osdmap e5: 0 total, 0 up, 0 in 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: mgrmap e15: smithi153.dolfed(active, starting, since 0.00352499s) 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr metadata", "who": "smithi153.dolfed", "id": "smithi153.dolfed"}]: dispatch 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T14:24:25.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: Manager daemon smithi153.dolfed is now available 2024-05-06T14:24:25.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:24:25.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:24:25.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 -- 172.21.15.190:0/4133817304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 msgr2=0x7f14c00ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:25.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 --2- 172.21.15.190:0/4133817304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c00ff140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009910 tx=0x7f14b4009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:25.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 -- 172.21.15.190:0/4133817304 shutdown_connections 2024-05-06T14:24:25.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 --2- 172.21.15.190:0/4133817304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c00ff140 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 -- 172.21.15.190:0/4133817304 >> 172.21.15.190:0/4133817304 conn(0x7f14c00fa2e0 msgr2=0x7f14c00fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 -- 172.21.15.190:0/4133817304 shutdown_connections 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.807+0000 7f14c6195700 1 -- 172.21.15.190:0/4133817304 wait complete. 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.808+0000 7f14c6195700 1 Processor -- start 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.808+0000 7f14c6195700 1 -- start start 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.808+0000 7f14c6195700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c0194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.808+0000 7f14c6195700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c0195500 con 0x7f14c00fed30 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c0194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c0194fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:50972/0 (socket says 172.21.15.190:50972) 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bf7fe700 1 -- 172.21.15.190:0/3998404814 learned_addr learned my addr 172.21.15.190:0/3998404814 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bf7fe700 1 -- 172.21.15.190:0/3998404814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40095f0 con 0x7f14c00fed30 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bf7fe700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c0194fc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f14b4005950 tx=0x7f14b4009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b401d070 con 0x7f14c00fed30 2024-05-06T14:24:25.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f14b400be10 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b400f460 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c0195700 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.809+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0195ba0 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.810+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) v1 ==== 44878+0+0 (secure 0 0 0) 0x7f14b400f5c0 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.810+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c0104b60 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.810+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f14b404c0d0 con 0x7f14c00fed30 2024-05-06T14:24:25.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.815+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f14b402a360 con 0x7f14c00fed30 2024-05-06T14:24:25.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.930+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 16) v1 ==== 45005+0+0 (secure 0 0 0) 0x7f14b40227c0 con 0x7f14c00fed30 2024-05-06T14:24:25.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.930+0000 7f14bcff9700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f14a8039520 0x7f14a803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:25.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.930+0000 7f14beffd700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f14a8039520 0x7f14a803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:25.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:25.931+0000 7f14beffd700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f14a8039520 0x7f14a803b900 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006fd0 tx=0x7f14b0006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:26.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.101+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14c0198980 con 0x7f14c00fed30 2024-05-06T14:24:26.089 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.101+0000 7f14bcff9700 1 -- 172.21.15.190:0/3998404814 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f14b4027030 con 0x7f14c00fed30 2024-05-06T14:24:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:26.089 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:26.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f14a8039520 msgr2=0x7f14a803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:26.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f14a8039520 0x7f14a803b900 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006fd0 tx=0x7f14b0006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:26.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 msgr2=0x7f14c0194fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:26.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c0194fc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f14b4005950 tx=0x7f14b4009e60 comp rx=0 tx=0).stop 2024-05-06T14:24:26.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 shutdown_connections 2024-05-06T14:24:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f14a8039520 0x7f14a803b900 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 --2- 172.21.15.190:0/3998404814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c00fed30 0x7f14c0194fc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 >> 172.21.15.190:0/3998404814 conn(0x7f14c00fa2e0 msgr2=0x7f14c00faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 shutdown_connections 2024-05-06T14:24:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:26.104+0000 7f14c6195700 1 -- 172.21.15.190:0/3998404814 wait complete. 2024-05-06T14:24:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:26.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:26.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:24:26.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/mirror_snapshot_schedule"}]: dispatch 2024-05-06T14:24:26.167 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.dolfed/trash_purge_schedule"}]: dispatch 2024-05-06T14:24:26.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:26.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:25 smithi153 ceph-mon[27177]: mgrmap e16: smithi153.dolfed(active, since 1.00825s) 2024-05-06T14:24:27.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:27 smithi153 ceph-mon[27177]: [06/May/2024:14:24:25] ENGINE Bus STARTING 2024-05-06T14:24:27.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:27 smithi153 ceph-mon[27177]: [06/May/2024:14:24:25] ENGINE Serving on https://172.21.15.153:7150 2024-05-06T14:24:27.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:27 smithi153 ceph-mon[27177]: [06/May/2024:14:24:25] ENGINE Serving on http://172.21.15.153:8765 2024-05-06T14:24:27.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:27 smithi153 ceph-mon[27177]: [06/May/2024:14:24:25] ENGINE Bus STARTED 2024-05-06T14:24:27.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:27 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:27.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:27 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/3998404814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:27.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:27.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:28.377 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:28 smithi153 ceph-mon[27177]: mgrmap e17: smithi153.dolfed(active, since 2s) 2024-05-06T14:24:28.378 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:28.839 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:28.895 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:30.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:24:30.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:24:30.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:30 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:24:31.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 -- 172.21.15.190:0/3226435628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 msgr2=0x7f58bc1048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:31.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 --2- 172.21.15.190:0/3226435628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc1048e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f58a400aa20 tx=0x7f58a400ad30 comp rx=0 tx=0).stop 2024-05-06T14:24:31.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 -- 172.21.15.190:0/3226435628 shutdown_connections 2024-05-06T14:24:31.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 --2- 172.21.15.190:0/3226435628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc1048e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:31.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 -- 172.21.15.190:0/3226435628 >> 172.21.15.190:0/3226435628 conn(0x7f58bc075780 msgr2=0x7f58bc075b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 -- 172.21.15.190:0/3226435628 shutdown_connections 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.236+0000 7f58c24fa700 1 -- 172.21.15.190:0/3226435628 wait complete. 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.237+0000 7f58c24fa700 1 Processor -- start 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.237+0000 7f58c24fa700 1 -- start start 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.237+0000 7f58c24fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.237+0000 7f58c24fa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1957a0 con 0x7f58bc104510 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.237+0000 7f58bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.237+0000 7f58bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc195260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:40704/0 (socket says 172.21.15.190:40704) 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58bbfff700 1 -- 172.21.15.190:0/1911442812 learned_addr learned my addr 172.21.15.190:0/1911442812 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58bbfff700 1 -- 172.21.15.190:0/1911442812 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a400a700 con 0x7f58bc104510 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58bbfff700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc195260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58a400cf80 tx=0x7f58a4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a401d070 con 0x7f58bc104510 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58a40053e0 con 0x7f58bc104510 2024-05-06T14:24:31.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58a400e600 con 0x7f58bc104510 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc195a00 con 0x7f58bc104510 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.238+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc195f00 con 0x7f58bc104510 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.239+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f58a400e870 con 0x7f58bc104510 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.239+0000 7f58b97fa700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f58a8038530 0x7f58a803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.239+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc062380 con 0x7f58bc104510 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.239+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f58a4012070 con 0x7f58bc104510 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.239+0000 7f58bb7fe700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f58a8038530 0x7f58a803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.240+0000 7f58bb7fe700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f58a8038530 0x7f58a803a9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58ac006fd0 tx=0x7f58ac006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:31.237 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.244+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f58a40153d0 con 0x7f58bc104510 2024-05-06T14:24:31.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.530+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58bc195f00 con 0x7f58bc104510 2024-05-06T14:24:31.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.531+0000 7f58b97fa700 1 -- 172.21.15.190:0/1911442812 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f58bc195f00 con 0x7f58bc104510 2024-05-06T14:24:31.518 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:31.519 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f58a8038530 msgr2=0x7f58a803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f58a8038530 0x7f58a803a9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58ac006fd0 tx=0x7f58ac006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 msgr2=0x7f58bc195260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc195260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f58a400cf80 tx=0x7f58a4004d10 comp rx=0 tx=0).stop 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 shutdown_connections 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f58a8038530 0x7f58a803a9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 --2- 172.21.15.190:0/1911442812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58bc104510 0x7f58bc195260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 >> 172.21.15.190:0/1911442812 conn(0x7f58bc075780 msgr2=0x7f58bc102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 shutdown_connections 2024-05-06T14:24:31.521 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:31.533+0000 7f58c24fa700 1 -- 172.21.15.190:0/1911442812 wait complete. 2024-05-06T14:24:31.522 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:31.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: Updating smithi153:/etc/ceph/ceph.conf 2024-05-06T14:24:31.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: Updating smithi190:/etc/ceph/ceph.conf 2024-05-06T14:24:31.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:31.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:31.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-06T14:24:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-05-06T14:24:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:31 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:24:32.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:24:32.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:24:32.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:24:32.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:24:32.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T14:24:32.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T14:24:32.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: Deploying daemon ceph-exporter.smithi190 on smithi190 2024-05-06T14:24:32.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:32 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1911442812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:33.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:33.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:33.549 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 -- 172.21.15.190:0/3079715888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 msgr2=0x7fa608100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 --2- 172.21.15.190:0/3079715888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608100480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009910 tx=0x7fa5f8009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 -- 172.21.15.190:0/3079715888 shutdown_connections 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 --2- 172.21.15.190:0/3079715888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608100480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 -- 172.21.15.190:0/3079715888 >> 172.21.15.190:0/3079715888 conn(0x7fa6080fb990 msgr2=0x7fa6080fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 -- 172.21.15.190:0/3079715888 shutdown_connections 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 -- 172.21.15.190:0/3079715888 wait complete. 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 Processor -- start 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.940+0000 7fa60e70f700 1 -- start start 2024-05-06T14:24:34.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.941+0000 7fa60e70f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:34.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.941+0000 7fa60e70f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608074fd0 con 0x7fa6081000b0 2024-05-06T14:24:34.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.941+0000 7fa607fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:34.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.941+0000 7fa607fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:40710/0 (socket says 172.21.15.190:40710) 2024-05-06T14:24:34.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.941+0000 7fa607fff700 1 -- 172.21.15.190:0/3194001891 learned_addr learned my addr 172.21.15.190:0/3194001891 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:34.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.941+0000 7fa607fff700 1 -- 172.21.15.190:0/3194001891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f80095f0 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.942+0000 7fa607fff700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608073050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8005b40 tx=0x7fa5f8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.942+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f801d070 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.942+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa608073590 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.942+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa5f80044d0 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.942+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f8003930 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.942+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa608073a10 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.943+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa608108330 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.943+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7fa5f800b960 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.943+0000 7fa6057fa700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa5f0038580 0x7fa5f003aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.943+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa5f804c4d0 con 0x7fa6081000b0 2024-05-06T14:24:34.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.943+0000 7fa6077fe700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa5f0038580 0x7fa5f003aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:34.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.944+0000 7fa6077fe700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa5f0038580 0x7fa5f003aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc006fd0 tx=0x7fa5fc006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:34.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:34.948+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa5f802a540 con 0x7fa6081000b0 2024-05-06T14:24:35.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.225+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa608073cb0 con 0x7fa6081000b0 2024-05-06T14:24:35.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.225+0000 7fa6057fa700 1 -- 172.21.15.190:0/3194001891 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa5f802a360 con 0x7fa6081000b0 2024-05-06T14:24:35.213 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:35.213 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa5f0038580 msgr2=0x7fa5f003aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa5f0038580 0x7fa5f003aa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc006fd0 tx=0x7fa5fc006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 msgr2=0x7fa608073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608073050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8005b40 tx=0x7fa5f8004c80 comp rx=0 tx=0).stop 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 shutdown_connections 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa5f0038580 0x7fa5f003aa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 --2- 172.21.15.190:0/3194001891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6081000b0 0x7fa608073050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 >> 172.21.15.190:0/3194001891 conn(0x7fa6080fb990 msgr2=0x7fa608104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 shutdown_connections 2024-05-06T14:24:35.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:35.228+0000 7fa60e70f700 1 -- 172.21.15.190:0/3194001891 wait complete. 2024-05-06T14:24:35.217 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:35.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:35 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/3194001891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-05-06T14:24:36.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:24:36.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:36 smithi153 ceph-mon[27177]: Deploying daemon crash.smithi190 on smithi190 2024-05-06T14:24:36.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:36.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:37.213 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 -- 172.21.15.190:0/3842288635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 msgr2=0x7f42c80ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 --2- 172.21.15.190:0/3842288635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c80ff3f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009910 tx=0x7f42c4009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 -- 172.21.15.190:0/3842288635 shutdown_connections 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 --2- 172.21.15.190:0/3842288635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c80ff3f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 -- 172.21.15.190:0/3842288635 >> 172.21.15.190:0/3842288635 conn(0x7f42c8074af0 msgr2=0x7f42c8074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 -- 172.21.15.190:0/3842288635 shutdown_connections 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.739+0000 7f42cfbd7700 1 -- 172.21.15.190:0/3842288635 wait complete. 2024-05-06T14:24:38.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cfbd7700 1 Processor -- start 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cfbd7700 1 -- start start 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cfbd7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c81961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cfbd7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8196710 con 0x7f42c80ff020 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cd973700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c81961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cd973700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c81961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:40742/0 (socket says 172.21.15.190:40742) 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.740+0000 7f42cd973700 1 -- 172.21.15.190:0/1559614655 learned_addr learned my addr 172.21.15.190:0/1559614655 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42cd973700 1 -- 172.21.15.190:0/1559614655 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40095f0 con 0x7f42c80ff020 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42cd973700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c81961d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005b40 tx=0x7f42c4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42c401d070 con 0x7f42c80ff020 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f42c40044d0 con 0x7f42c80ff020 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c8196910 con 0x7f42c80ff020 2024-05-06T14:24:38.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42c4003930 con 0x7f42c80ff020 2024-05-06T14:24:38.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.741+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c8196cd0 con 0x7f42c80ff020 2024-05-06T14:24:38.733 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.742+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f42c4003bf0 con 0x7f42c80ff020 2024-05-06T14:24:38.733 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.742+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c8107250 con 0x7f42c80ff020 2024-05-06T14:24:38.733 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.742+0000 7f42beffd700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f42b4038580 0x7f42b403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:38.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.742+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f42c404c6d0 con 0x7f42c80ff020 2024-05-06T14:24:38.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.742+0000 7f42cd172700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f42b4038580 0x7f42b403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:38.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.743+0000 7f42cd172700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f42b4038580 0x7f42b403aa30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f42b8006fd0 tx=0x7f42b8006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:38.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:38.747+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f42c400f4c0 con 0x7f42c80ff020 2024-05-06T14:24:39.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.017+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42c8068550 con 0x7f42c80ff020 2024-05-06T14:24:39.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.018+0000 7f42beffd700 1 -- 172.21.15.190:0/1559614655 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f42c4022e60 con 0x7f42c80ff020 2024-05-06T14:24:39.007 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:39.007 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f42b4038580 msgr2=0x7f42b403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f42b4038580 0x7f42b403aa30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f42b8006fd0 tx=0x7f42b8006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 msgr2=0x7f42c81961d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c81961d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005b40 tx=0x7f42c4004c80 comp rx=0 tx=0).stop 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 shutdown_connections 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f42b4038580 0x7f42b403aa30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 --2- 172.21.15.190:0/1559614655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c80ff020 0x7f42c81961d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 >> 172.21.15.190:0/1559614655 conn(0x7f42c8074af0 msgr2=0x7f42c81031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 shutdown_connections 2024-05-06T14:24:39.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:39.020+0000 7f42cfbd7700 1 -- 172.21.15.190:0/1559614655 wait complete. 2024-05-06T14:24:39.010 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:40.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:39 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1559614655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:40.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:40.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:40.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:40.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:40.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:39 smithi153 ceph-mon[27177]: Deploying daemon node-exporter.smithi190 on smithi190 2024-05-06T14:24:40.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:40.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:40.992 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:24:41.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:41.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:24:42.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 -- 172.21.15.190:0/426994535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 msgr2=0x7f2c740ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:42.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 --2- 172.21.15.190:0/426994535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c740ff3f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009910 tx=0x7f2c64009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 -- 172.21.15.190:0/426994535 shutdown_connections 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 --2- 172.21.15.190:0/426994535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c740ff3f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 -- 172.21.15.190:0/426994535 >> 172.21.15.190:0/426994535 conn(0x7f2c74074af0 msgr2=0x7f2c74074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 -- 172.21.15.190:0/426994535 shutdown_connections 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 -- 172.21.15.190:0/426994535 wait complete. 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.905+0000 7f2c7a480700 1 Processor -- start 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c7a480700 1 -- start start 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c7a480700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c741961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c7a480700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74196710 con 0x7f2c740ff020 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c73fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c741961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c73fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c741961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:45544/0 (socket says 172.21.15.190:45544) 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c73fff700 1 -- 172.21.15.190:0/893711363 learned_addr learned my addr 172.21.15.190:0/893711363 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:42.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.906+0000 7f2c73fff700 1 -- 172.21.15.190:0/893711363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c640095f0 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.907+0000 7f2c73fff700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c741961d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c64005b40 tx=0x7f2c64004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.907+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6401d070 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.907+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2c640044d0 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.907+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c74196910 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.907+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c64003930 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.907+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c74196cd0 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.908+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c74107250 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.908+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f2c6400b960 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.908+0000 7f2c717fa700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2c5c038580 0x7f2c5c03aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.908+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f2c6404c470 con 0x7f2c740ff020 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.908+0000 7f2c737fe700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2c5c038580 0x7f2c5c03aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.909+0000 7f2c737fe700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2c5c038580 0x7f2c5c03aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2c68006fd0 tx=0x7f2c68006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:42.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:42.913+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2c6402a540 con 0x7f2c740ff020 2024-05-06T14:24:43.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.224+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c74068550 con 0x7f2c740ff020 2024-05-06T14:24:43.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.225+0000 7f2c717fa700 1 -- 172.21.15.190:0/893711363 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2c6402a360 con 0x7f2c740ff020 2024-05-06T14:24:43.214 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:43.214 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2c5c038580 msgr2=0x7f2c5c03aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2c5c038580 0x7f2c5c03aa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2c68006fd0 tx=0x7f2c68006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 msgr2=0x7f2c741961d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c741961d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c64005b40 tx=0x7f2c64004c80 comp rx=0 tx=0).stop 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 shutdown_connections 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2c5c038580 0x7f2c5c03aa30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 --2- 172.21.15.190:0/893711363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c740ff020 0x7f2c741961d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 >> 172.21.15.190:0/893711363 conn(0x7f2c74074af0 msgr2=0x7f2c741031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 shutdown_connections 2024-05-06T14:24:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:43.227+0000 7f2c7a480700 1 -- 172.21.15.190:0/893711363 wait complete. 2024-05-06T14:24:43.217 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:43.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:43 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/893711363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:45.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:45.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:45.926 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.qauohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.qauohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T14:24:46.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:24:46.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:24:46.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: Deploying daemon mgr.smithi190.qauohf on smithi190 2024-05-06T14:24:46.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:46.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:48.155 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 -- 172.21.15.190:0/3845506664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 msgr2=0x7f0eb8101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 --2- 172.21.15.190:0/3845506664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb8101360 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009910 tx=0x7f0ea8009c20 comp rx=0 tx=0).stop 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 -- 172.21.15.190:0/3845506664 shutdown_connections 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 --2- 172.21.15.190:0/3845506664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb8101360 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 -- 172.21.15.190:0/3845506664 >> 172.21.15.190:0/3845506664 conn(0x7f0eb80fa900 msgr2=0x7f0eb80fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 -- 172.21.15.190:0/3845506664 shutdown_connections 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.166+0000 7f0ebc82e700 1 -- 172.21.15.190:0/3845506664 wait complete. 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.167+0000 7f0ebc82e700 1 Processor -- start 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.167+0000 7f0ebc82e700 1 -- start start 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.167+0000 7f0ebc82e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb81953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:48.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.167+0000 7f0ebc82e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb8195920 con 0x7f0eb8100f90 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.167+0000 7f0eb659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb81953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.167+0000 7f0eb659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb81953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.190:45564/0 (socket says 172.21.15.190:45564) 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0eb659c700 1 -- 172.21.15.190:0/2789182177 learned_addr learned my addr 172.21.15.190:0/2789182177 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0eb659c700 1 -- 172.21.15.190:0/2789182177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea80095f0 con 0x7f0eb8100f90 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0eb659c700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb81953e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8005b40 tx=0x7f0ea8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ea801d070 con 0x7f0eb8100f90 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0ea80044d0 con 0x7f0eb8100f90 2024-05-06T14:24:48.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb8195b20 con 0x7f0eb8100f90 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ea8003930 con 0x7f0eb8100f90 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.168+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb8195ee0 con 0x7f0eb8100f90 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.169+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb8062380 con 0x7f0eb8100f90 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.169+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 45199+0+0 (secure 0 0 0) 0x7f0ea800b960 con 0x7f0eb8100f90 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.169+0000 7f0ea77fe700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f0ea0038580 0x7f0ea003aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.169+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f0ea804c4b0 con 0x7f0eb8100f90 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.170+0000 7f0eb5d9b700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f0ea0038580 0x7f0ea003aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:48.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.170+0000 7f0eb5d9b700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f0ea0038580 0x7f0ea003aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0eac006fd0 tx=0x7f0eac006e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:48.163 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.174+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0ea802a540 con 0x7f0eb8100f90 2024-05-06T14:24:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:47 smithi153 ceph-mon[27177]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:48.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.464+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0eb8107120 con 0x7f0eb8100f90 2024-05-06T14:24:48.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.464+0000 7f0ea77fe700 1 -- 172.21.15.190:0/2789182177 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f0ea802a360 con 0x7f0eb8100f90 2024-05-06T14:24:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:48.453 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:22:09.924627Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f0ea0038580 msgr2=0x7f0ea003aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f0ea0038580 0x7f0ea003aa30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0eac006fd0 tx=0x7f0eac006e40 comp rx=0 tx=0).stop 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 msgr2=0x7f0eb81953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb81953e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8005b40 tx=0x7f0ea8004c80 comp rx=0 tx=0).stop 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 shutdown_connections 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f0ea0038580 0x7f0ea003aa30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 --2- 172.21.15.190:0/2789182177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb8100f90 0x7f0eb81953e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.466+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 >> 172.21.15.190:0/2789182177 conn(0x7f0eb80fa900 msgr2=0x7f0eb81047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:48.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.467+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 shutdown_connections 2024-05-06T14:24:48.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:48.467+0000 7f0ebc82e700 1 -- 172.21.15.190:0/2789182177 wait complete. 2024-05-06T14:24:48.456 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-05-06T14:24:49.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/2789182177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:24:49.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:49.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:49.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:49.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:49.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T14:24:49.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:24:50.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:49 smithi153 ceph-mon[27177]: Deploying daemon mon.smithi190 on smithi190 2024-05-06T14:24:50.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:49 smithi153 ceph-mon[27177]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:50.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:49 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:50.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-05-06T14:24:50.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T14:24:51.617 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:24:52.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:51 smithi153 ceph-mon[27177]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:58.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:24:58.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: mon.smithi153 calling monitor election 2024-05-06T14:24:58.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:58.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.190:0/2955354452' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/crt"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: mon.smithi190 calling monitor election 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: mon.smithi153 is new leader, mons smithi153,smithi190 in quorum (ranks 0,1) 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi190=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: fsmap 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: osdmap e5: 0 total, 0 up, 0 in 2024-05-06T14:24:58.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: mgrmap e17: smithi153.dolfed(active, since 33s) 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: Standby manager daemon smithi190.qauohf started 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.190:0/2955354452' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: overall HEALTH_OK 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.190:0/2955354452' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/key"}]: dispatch 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.190:0/2955354452' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:58.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:58 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:58.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:58 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:59.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/483651034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 msgr2=0x7fd3c000a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:59.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/483651034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3c000a8a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8100000 tx=0x7fd3c800fe90 comp rx=0 tx=0).stop 2024-05-06T14:24:59.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/483651034 shutdown_connections 2024-05-06T14:24:59.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/483651034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3c000a8a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:59.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/483651034 >> 172.21.15.190:0/483651034 conn(0x7fd3d8074c10 msgr2=0x7fd3d8075010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:59.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/483651034 shutdown_connections 2024-05-06T14:24:59.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.693+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/483651034 wait complete. 2024-05-06T14:24:59.683 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:59.683 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: mgrmap e18: smithi153.dolfed(active, since 33s), standbys: smithi190.qauohf 2024-05-06T14:24:59.684 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr metadata", "who": "smithi190.qauohf", "id": "smithi190.qauohf"}]: dispatch 2024-05-06T14:24:59.684 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:59.684 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:24:59.684 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:59.684 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:24:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:59.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.694+0000 7fd3ddbf0700 1 Processor -- start 2024-05-06T14:24:59.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.694+0000 7fd3ddbf0700 1 -- start start 2024-05-06T14:24:59.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3ddbf0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3d819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:59.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3ddbf0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:59.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3ddbf0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d819e070 con 0x7fd3c0008440 2024-05-06T14:24:59.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3ddbf0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8197b30 con 0x7fd3d819dab0 2024-05-06T14:24:59.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:58328/0 (socket says 172.21.15.190:58328) 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 -- 172.21.15.190:0/1234981503 learned_addr learned my addr 172.21.15.190:0/1234981503 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 -- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 msgr2=0x7fd3d81975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 -- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 msgr2=0x7fd3d81975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d6ffd700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d77fe700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3d819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d77fe700 1 -- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 msgr2=0x7fd3d81975f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d77fe700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.695+0000 7fd3d77fe700 1 -- 172.21.15.190:0/1234981503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c800f8e0 con 0x7fd3c0008440 2024-05-06T14:24:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.696+0000 7fd3d77fe700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3d819d570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8100000 tx=0x7fd3c801d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.696+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c8005da0 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.696+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d8197db0 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.696+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d8198300 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.696+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd3c801dc10 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.696+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c8011730 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.697+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d8062380 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.697+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7fd3c8011890 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.698+0000 7fd3d4ff9700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd3c406c700 0x7fd3c406ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.698+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fd3c808d740 con 0x7fd3c0008440 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.698+0000 7fd3d6ffd700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd3c406c700 0x7fd3c406ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:24:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.698+0000 7fd3d6ffd700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd3c406c700 0x7fd3c406ebb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009180 tx=0x7fd3cc008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:24:59.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.703+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd3c805c9a0 con 0x7fd3c0008440 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: mgrmap e18: smithi153.dolfed(active, since 33s), standbys: smithi190.qauohf 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr metadata", "who": "smithi190.qauohf", "id": "smithi190.qauohf"}]: dispatch 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:24:59.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:24:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:24:59.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.973+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3d8105d40 con 0x7fd3c0008440 2024-05-06T14:24:59.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.974+0000 7fd3d4ff9700 1 -- 172.21.15.190:0/1234981503 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7fd3c8035420 con 0x7fd3c0008440 2024-05-06T14:24:59.962 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-05-06T14:24:59.962 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","modified":"2024-05-06T14:24:53.426433Z","created":"2024-05-06T14:22:09.924627Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi190","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd3c406c700 msgr2=0x7fd3c406ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd3c406c700 0x7fd3c406ebb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009180 tx=0x7fd3cc008040 comp rx=0 tx=0).stop 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 msgr2=0x7fd3d819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3d819d570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8100000 tx=0x7fd3c801d9b0 comp rx=0 tx=0).stop 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 shutdown_connections 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3c0008440 0x7fd3d819d570 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd3c406c700 0x7fd3c406ebb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 --2- 172.21.15.190:0/1234981503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3d819dab0 0x7fd3d81975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:24:59.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.976+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 >> 172.21.15.190:0/1234981503 conn(0x7fd3d8074c10 msgr2=0x7fd3d80fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:24:59.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.977+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 shutdown_connections 2024-05-06T14:24:59.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:24:59.977+0000 7fd3ddbf0700 1 -- 172.21.15.190:0/1234981503 wait complete. 2024-05-06T14:24:59.965 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2024-05-06T14:25:00.593 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:00 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/1234981503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:25:00.593 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:00 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:00.617 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-06T14:25:00.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-06T14:25:00.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:00 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1234981503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T14:25:00.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:00 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:00.831 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:25:02.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:02.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:02.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: Updating smithi153:/etc/ceph/ceph.conf 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: Updating smithi190:/etc/ceph/ceph.conf 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T14:25:02.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T14:25:02.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:02.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:02.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: Updating smithi153:/etc/ceph/ceph.conf 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: Updating smithi190:/etc/ceph/ceph.conf 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:02.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T14:25:02.438 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T14:25:02.438 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:03.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:03 smithi153 ceph-mon[27177]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-05-06T14:25:03.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:03 smithi153 ceph-mon[27177]: Reconfiguring daemon mon.smithi153 on smithi153 2024-05-06T14:25:03.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 -- 172.21.15.153:0/2123560639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ffc80 msgr2=0x7fb7ac103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/2123560639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ffc80 0x7fb7ac103c50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb794009910 tx=0x7fb794009c20 comp rx=0 tx=0).stop 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 -- 172.21.15.153:0/2123560639 shutdown_connections 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/2123560639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ffc80 0x7fb7ac103c50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/2123560639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 -- 172.21.15.153:0/2123560639 >> 172.21.15.153:0/2123560639 conn(0x7fb7ac0fab80 msgr2=0x7fb7ac0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 -- 172.21.15.153:0/2123560639 shutdown_connections 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.431+0000 7fb7b1bad700 1 -- 172.21.15.153:0/2123560639 wait complete. 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7b1bad700 1 Processor -- start 2024-05-06T14:25:03.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7b1bad700 1 -- start start 2024-05-06T14:25:03.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7b1bad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:03.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7b1bad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ac0ffc80 0x7fb7ac101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:03.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7b1bad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac101770 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7b1bad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1018b0 con 0x7fb7ac0ffc80 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7ab7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7ab7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37628/0 (socket says 172.21.15.153:37628) 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7ab7fe700 1 -- 172.21.15.153:0/1118038182 learned_addr learned my addr 172.21.15.153:0/1118038182 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.432+0000 7fb7a2ffd700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ac0ffc80 0x7fb7ac101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7ab7fe700 1 -- 172.21.15.153:0/1118038182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ac0ffc80 msgr2=0x7fb7ac101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7ab7fe700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ac0ffc80 0x7fb7ac101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7ab7fe700 1 -- 172.21.15.153:0/1118038182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940095f0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7ab7fe700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac100cf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00c8a0 tx=0x7fb79c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb79c0078e0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb79c00f450 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac1a1ee0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb79c00e5e0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.433+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1a22f0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.434+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac04ea50 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.435+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7fb79c00f5c0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.435+0000 7fb7a97fa700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb79806c600 0x7fb79806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:03.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.436+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb79c08a5f0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.436+0000 7fb7a2ffd700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb79806c600 0x7fb79806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:03.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.436+0000 7fb7a2ffd700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb79806c600 0x7fb79806eab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac1024e0 tx=0x7fb79401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:25:03.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.442+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb79c055e30 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:03 smithi190 ceph-mon[33671]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-05-06T14:25:03.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:03 smithi190 ceph-mon[33671]: Reconfiguring daemon mon.smithi153 on smithi153 2024-05-06T14:25:03.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.639+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb7ac1a2590 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.639+0000 7fb7a97fa700 1 -- 172.21.15.153:0/1118038182 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7fb79c089db0 con 0x7fb7ac0ff2e0 2024-05-06T14:25:03.627 INFO:teuthology.orchestra.run.smithi153.stdout:# minimal ceph.conf for eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:25:03.628 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-05-06T14:25:03.628 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:25:03.628 INFO:teuthology.orchestra.run.smithi153.stdout: mon_host = [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-05-06T14:25:03.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.641+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb79806c600 msgr2=0x7fb79806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.641+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb79806c600 0x7fb79806eab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac1024e0 tx=0x7fb79401a040 comp rx=0 tx=0).stop 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.641+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 msgr2=0x7fb7ac100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.641+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac100cf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00c8a0 tx=0x7fb79c00cbb0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 shutdown_connections 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ac0ff2e0 0x7fb7ac100cf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb79806c600 0x7fb79806eab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 --2- 172.21.15.153:0/1118038182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7ac0ffc80 0x7fb7ac101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 >> 172.21.15.153:0/1118038182 conn(0x7fb7ac0fab80 msgr2=0x7fb7ac0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 shutdown_connections 2024-05-06T14:25:03.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:03.642+0000 7fb7b1bad700 1 -- 172.21.15.153:0/1118038182 wait complete. 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: Reconfiguring mgr.smithi153.dolfed (unknown last config time)... 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.dolfed", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:04.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: Reconfiguring daemon mgr.smithi153.dolfed on smithi153 2024-05-06T14:25:04.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:04 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1118038182' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:04.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:04.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:04.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:04.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: Reconfiguring mgr.smithi153.dolfed (unknown last config time)... 2024-05-06T14:25:04.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.dolfed", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:25:04.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:25:04.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:04.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: Reconfiguring daemon mgr.smithi153.dolfed on smithi153 2024-05-06T14:25:04.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:04 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1118038182' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:04.655 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-06T14:25:04.655 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:25:04.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-05-06T14:25:04.681 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:25:04.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:25:04.745 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:25:04.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-05-06T14:25:04.777 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:25:04.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:25:04.844 INFO:tasks.cephadm:Deploying OSDs... 2024-05-06T14:25:04.844 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:25:04.844 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T14:25:04.862 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T14:25:04.862 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-05-06T14:25:04.916 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T14:25:04.916 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-05-06 14:24:16.721564445 +0000 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-05-06 14:18:50.669670118 +0000 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-05-06 14:18:50.669670118 +0000 2024-05-06T14:25:04.917 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-05-06 14:18:50.669670118 +0000 2024-05-06T14:25:04.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T14:25:04.980 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-05-06T14:25:04.980 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-05-06T14:25:04.980 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000208725 s, 2.5 MB/s 2024-05-06T14:25:04.981 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T14:25:05.037 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-05-06T14:25:05.096 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-05-06 14:24:16.721564445 +0000 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-05-06 14:18:51.195659279 +0000 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-05-06 14:18:51.195659279 +0000 2024-05-06T14:25:05.097 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-05-06 14:18:51.195659279 +0000 2024-05-06T14:25:05.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T14:25:05.163 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-05-06T14:25:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-05-06T14:25:05.164 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000185688 s, 2.8 MB/s 2024-05-06T14:25:05.165 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T14:25:05.226 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-05-06 14:24:16.721564445 +0000 2024-05-06T14:25:05.287 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-05-06 14:18:51.517652643 +0000 2024-05-06T14:25:05.288 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-05-06 14:18:51.517652643 +0000 2024-05-06T14:25:05.288 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-05-06 14:18:51.517652643 +0000 2024-05-06T14:25:05.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T14:25:05.361 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-05-06T14:25:05.361 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-05-06T14:25:05.361 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000205077 s, 2.5 MB/s 2024-05-06T14:25:05.363 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T14:25:05.430 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-05-06 14:24:16.722564425 +0000 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-05-06 14:18:51.851645760 +0000 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-05-06 14:18:51.851645760 +0000 2024-05-06T14:25:05.490 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-05-06 14:18:51.851645760 +0000 2024-05-06T14:25:05.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T14:25:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-05-06T14:25:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-05-06T14:25:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000197593 s, 2.6 MB/s 2024-05-06T14:25:05.555 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T14:25:05.616 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:25:05.616 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T14:25:05.637 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T14:25:05.637 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-05-06 14:24:51.788561096 +0000 2024-05-06T14:25:05.697 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-05-06 14:18:58.496408592 +0000 2024-05-06T14:25:05.698 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-05-06 14:18:58.496408592 +0000 2024-05-06T14:25:05.698 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-05-06 14:18:58.496408592 +0000 2024-05-06T14:25:05.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T14:25:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-05-06T14:25:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-05-06T14:25:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00022179 s, 2.3 MB/s 2024-05-06T14:25:05.763 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T14:25:05.822 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-05-06T14:25:05.878 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T14:25:05.878 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:05.878 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-06T14:25:05.879 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:05.879 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:05.879 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-05-06 14:24:51.789561068 +0000 2024-05-06T14:25:05.879 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-05-06 14:18:58.837398498 +0000 2024-05-06T14:25:05.879 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-05-06 14:18:58.837398498 +0000 2024-05-06T14:25:05.879 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-05-06 14:18:58.837398498 +0000 2024-05-06T14:25:05.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T14:25:05.941 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-05-06T14:25:05.942 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-05-06T14:25:05.942 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000228845 s, 2.2 MB/s 2024-05-06T14:25:05.943 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T14:25:05.999 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-05-06T14:25:06.057 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T14:25:06.057 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:06.057 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-06T14:25:06.057 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:06.057 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:06.057 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-05-06 14:24:51.789561068 +0000 2024-05-06T14:25:06.058 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-05-06 14:18:59.174388522 +0000 2024-05-06T14:25:06.058 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-05-06 14:18:59.174388522 +0000 2024-05-06T14:25:06.058 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-05-06 14:18:59.174388522 +0000 2024-05-06T14:25:06.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T14:25:06.120 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-05-06T14:25:06.120 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-05-06T14:25:06.120 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000249313 s, 2.1 MB/s 2024-05-06T14:25:06.122 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T14:25:06.178 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-05-06 14:24:51.790561038 +0000 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-05-06 14:18:59.506378694 +0000 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-05-06 14:18:59.506378694 +0000 2024-05-06T14:25:06.234 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-05-06 14:18:59.506378694 +0000 2024-05-06T14:25:06.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: Reconfiguring ceph-exporter.smithi153 (monmap changed)... 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:06 smithi153 ceph-mon[27177]: Reconfiguring daemon ceph-exporter.smithi153 on smithi153 2024-05-06T14:25:06.296 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-05-06T14:25:06.296 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-05-06T14:25:06.296 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000218566 s, 2.3 MB/s 2024-05-06T14:25:06.297 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T14:25:06.353 INFO:tasks.cephadm:Deploying osd.0 on smithi153 with /dev/vg_nvme/lv_4... 2024-05-06T14:25:06.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-06T14:25:06.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:06.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:06.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:06.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: Reconfiguring ceph-exporter.smithi153 (monmap changed)... 2024-05-06T14:25:06.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi153", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-06T14:25:06.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:06.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:06 smithi190 ceph-mon[33671]: Reconfiguring daemon ceph-exporter.smithi153 on smithi153 2024-05-06T14:25:06.661 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:25:07.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:07.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:07.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: Reconfiguring crash.smithi153 (monmap changed)... 2024-05-06T14:25:07.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-06T14:25:07.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:07.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: Reconfiguring daemon crash.smithi153 on smithi153 2024-05-06T14:25:07.900 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:07 smithi153 ceph-mon[27177]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:07.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:07.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:07.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: Reconfiguring crash.smithi153 (monmap changed)... 2024-05-06T14:25:07.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-06T14:25:07.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:07.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: Reconfiguring daemon crash.smithi153 on smithi153 2024-05-06T14:25:07.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:07 smithi190 ceph-mon[33671]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-06T14:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-06T14:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-05-06T14:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-05-06T14:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117936 s, 889 MB/s 2024-05-06T14:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-05-06T14:25:09.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-05-06T14:25:09.790 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:25:10.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:09 smithi153 ceph-mon[27177]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:10.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:09 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:10.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:09 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:10.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:09 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:10.389 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:09 smithi190 ceph-mon[33671]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:10.390 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:09 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:10.390 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:09 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:10.390 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:09 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:11.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:10 smithi153 ceph-mon[27177]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-05-06T14:25:11.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:10 smithi153 ceph-mon[27177]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-05-06T14:25:11.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:10 smithi190 ceph-mon[33671]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-05-06T14:25:11.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:10 smithi190 ceph-mon[33671]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-05-06T14:25:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 -- 172.21.15.153:0/1848634244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 msgr2=0x7f92bc102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1848634244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc102fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009910 tx=0x7f92ac009c20 comp rx=0 tx=0).stop 2024-05-06T14:25:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 -- 172.21.15.153:0/1848634244 shutdown_connections 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1848634244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92bc103500 0x7f92bc1058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1848634244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc102fc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 -- 172.21.15.153:0/1848634244 >> 172.21.15.153:0/1848634244 conn(0x7f92bc0fa600 msgr2=0x7f92bc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 -- 172.21.15.153:0/1848634244 shutdown_connections 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 -- 172.21.15.153:0/1848634244 wait complete. 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.612+0000 7f92c1da2700 1 Processor -- start 2024-05-06T14:25:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92c1da2700 1 -- start start 2024-05-06T14:25:11.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92c1da2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92c1da2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92bc103500 0x7f92bc195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92c1da2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc1964e0 con 0x7f92bc100be0 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92c1da2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc196620 con 0x7f92bc103500 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92bb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92bb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47202/0 (socket says 172.21.15.153:47202) 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92bb7fe700 1 -- 172.21.15.153:0/1675821207 learned_addr learned my addr 172.21.15.153:0/1675821207 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92b2ffd700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92bc103500 0x7f92bc195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92bb7fe700 1 -- 172.21.15.153:0/1675821207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92bc103500 msgr2=0x7f92bc195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92bb7fe700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92bc103500 0x7f92bc195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.613+0000 7f92bb7fe700 1 -- 172.21.15.153:0/1675821207 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac0095f0 con 0x7f92bc100be0 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.614+0000 7f92bb7fe700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc195980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00b5c0 tx=0x7f92ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:25:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.614+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac01d070 con 0x7f92bc100be0 2024-05-06T14:25:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.614+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f92ac022470 con 0x7f92bc100be0 2024-05-06T14:25:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.614+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc19b020 con 0x7f92bc100be0 2024-05-06T14:25:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.614+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac00f690 con 0x7f92bc100be0 2024-05-06T14:25:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.614+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc19b510 con 0x7f92bc100be0 2024-05-06T14:25:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.615+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc18fb70 con 0x7f92bc100be0 2024-05-06T14:25:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.615+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f92ac004b70 con 0x7f92bc100be0 2024-05-06T14:25:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.616+0000 7f92b97fa700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f92a806c610 0x7f92a806eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.616+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f92ac08bda0 con 0x7f92bc100be0 2024-05-06T14:25:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.616+0000 7f92b2ffd700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f92a806c610 0x7f92a806eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.616+0000 7f92b2ffd700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f92a806c610 0x7f92a806eac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f92a4005fd0 tx=0x7f92a400a560 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:25:11.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.620+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f92ac05b120 con 0x7f92bc100be0 2024-05-06T14:25:11.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:11.826+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f92bc061190 con 0x7f92a806c610 2024-05-06T14:25:12.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:11 smithi153 ceph-mon[27177]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:25:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:25:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:12.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:11 smithi190 ceph-mon[33671]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:12.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:25:12.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:25:12.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:13.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:12 smithi153 ceph-mon[27177]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:25:13.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:12 smithi190 ceph-mon[33671]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:25:14.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:13 smithi153 ceph-mon[27177]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:14.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:13 smithi190 ceph-mon[33671]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:16.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:16 smithi153 ceph-mon[27177]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:16.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:16 smithi190 ceph-mon[33671]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:17 smithi153 ceph-mon[27177]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-05-06T14:25:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:17 smithi153 ceph-mon[27177]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-05-06T14:25:18.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:17 smithi153 ceph-mon[27177]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:18.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:18.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:18.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:17 smithi190 ceph-mon[33671]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-05-06T14:25:18.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:17 smithi190 ceph-mon[33671]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-05-06T14:25:18.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:17 smithi190 ceph-mon[33671]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:19.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:18 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/423253635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de83eb29-8176-4eb5-9be6-64b9e5449892"}]: dispatch 2024-05-06T14:25:19.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:18 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/423253635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de83eb29-8176-4eb5-9be6-64b9e5449892"}]': finished 2024-05-06T14:25:19.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:18 smithi153 ceph-mon[27177]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T14:25:19.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:18 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:19.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:18 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/52984043' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:25:19.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:18 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/423253635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de83eb29-8176-4eb5-9be6-64b9e5449892"}]: dispatch 2024-05-06T14:25:19.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:18 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/423253635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de83eb29-8176-4eb5-9be6-64b9e5449892"}]': finished 2024-05-06T14:25:19.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:18 smithi190 ceph-mon[33671]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T14:25:19.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:18 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:19.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:18 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/52984043' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:25:20.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:19 smithi153 ceph-mon[27177]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:20.402 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:19 smithi190 ceph-mon[33671]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:22.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:21 smithi153 ceph-mon[27177]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:22.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:21 smithi190 ceph-mon[33671]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:24.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:23 smithi153 ceph-mon[27177]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:23 smithi190 ceph-mon[33671]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:25.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:25.409 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:26.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:25 smithi153 ceph-mon[27177]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:26.440 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:25 smithi190 ceph-mon[33671]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:28.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:27 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:28.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:27 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:28.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:27 smithi190 ceph-mon[33671]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:28.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:27 smithi190 ceph-mon[33671]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-05-06T14:25:28.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:27 smithi190 ceph-mon[33671]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-05-06T14:25:28.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:27 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:28.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:27 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:28.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:27 smithi153 ceph-mon[27177]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:28.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:27 smithi153 ceph-mon[27177]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-05-06T14:25:28.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:27 smithi153 ceph-mon[27177]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-05-06T14:25:29.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T14:25:29.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:29.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:28 smithi190 ceph-mon[33671]: Deploying daemon osd.0 on smithi153 2024-05-06T14:25:29.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T14:25:29.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:29.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:28 smithi153 ceph-mon[27177]: Deploying daemon osd.0 on smithi153 2024-05-06T14:25:30.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:29 smithi153 ceph-mon[27177]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:30.415 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:29 smithi190 ceph-mon[33671]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:32.126 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:31 smithi153 ceph-mon[27177]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:32.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:31 smithi190 ceph-mon[33671]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:33.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:33.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:33.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: Reconfiguring ceph-exporter.smithi190 (monmap changed)... 2024-05-06T14:25:33.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-06T14:25:33.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:33.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: Reconfiguring daemon ceph-exporter.smithi190 on smithi190 2024-05-06T14:25:33.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:33 smithi153 ceph-mon[27177]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:33.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:33.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:33.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: Reconfiguring ceph-exporter.smithi190 (monmap changed)... 2024-05-06T14:25:33.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi190", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-05-06T14:25:33.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:33.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: Reconfiguring daemon ceph-exporter.smithi190 on smithi190 2024-05-06T14:25:33.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:33 smithi190 ceph-mon[33671]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:35.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:35.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:35.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:35 smithi153 ceph-mon[27177]: Reconfiguring crash.smithi190 (monmap changed)... 2024-05-06T14:25:35.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-06T14:25:35.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:35.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:35 smithi153 ceph-mon[27177]: Reconfiguring daemon crash.smithi190 on smithi190 2024-05-06T14:25:35.422 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:35.422 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:35.422 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:35 smithi190 ceph-mon[33671]: Reconfiguring crash.smithi190 (monmap changed)... 2024-05-06T14:25:35.422 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi190", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-05-06T14:25:35.422 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:35.422 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:35 smithi190 ceph-mon[33671]: Reconfiguring daemon crash.smithi190 on smithi190 2024-05-06T14:25:36.326 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:36 smithi190 ceph-mon[33671]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:36.326 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:36 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:36.326 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:36 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:36.326 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:36 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.qauohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:25:36.327 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:36 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:25:36.327 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:36 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:36.434 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:36 smithi153 ceph-mon[27177]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:36.434 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:36.434 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:36.434 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.qauohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:25:36.435 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:25:36.435 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:37.378 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: Reconfiguring mgr.smithi190.qauohf (monmap changed)... 2024-05-06T14:25:37.378 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: Reconfiguring daemon mgr.smithi190.qauohf on smithi190 2024-05-06T14:25:37.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:37.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:37.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T14:25:37.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T14:25:37.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: Reconfiguring mgr.smithi190.qauohf (monmap changed)... 2024-05-06T14:25:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: Reconfiguring daemon mgr.smithi190.qauohf on smithi190 2024-05-06T14:25:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:37.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:37.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T14:25:37.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T14:25:37.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:38.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:38 smithi153 ceph-mon[27177]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:38.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:38 smithi153 ceph-mon[27177]: Reconfiguring mon.smithi190 (monmap changed)... 2024-05-06T14:25:38.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:38 smithi153 ceph-mon[27177]: Reconfiguring daemon mon.smithi190 on smithi190 2024-05-06T14:25:38.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:38 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:38.335 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:38 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:38 smithi190 ceph-mon[33671]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:38 smithi190 ceph-mon[33671]: Reconfiguring mon.smithi190 (monmap changed)... 2024-05-06T14:25:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:38 smithi190 ceph-mon[33671]: Reconfiguring daemon mon.smithi190 on smithi190 2024-05-06T14:25:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:38 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:38 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T14:25:39.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153:9093"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153:9093"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153:3000"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153:3000"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153:9095"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153:9095"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:25:39.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:39 smithi190 ceph-mon[33671]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:39.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153:9093"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153:9093"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153:3000"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153:3000"}]: dispatch 2024-05-06T14:25:39.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153:9095"}]: dispatch 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153:9095"}]: dispatch 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:25:39.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:39 smithi153 ceph-mon[27177]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:40.588 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:40.589 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:40.589 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:40.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.633+0000 7f92b97fa700 1 -- 172.21.15.153:0/1675821207 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f92bc061190 con 0x7f92a806c610 2024-05-06T14:25:40.622 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 0 on host 'smithi153' 2024-05-06T14:25:40.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f92a806c610 msgr2=0x7f92a806eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:40.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f92a806c610 0x7f92a806eac0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f92a4005fd0 tx=0x7f92a400a560 comp rx=0 tx=0).stop 2024-05-06T14:25:40.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 msgr2=0x7f92bc195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:40.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc195980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00b5c0 tx=0x7f92ac00be20 comp rx=0 tx=0).stop 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 shutdown_connections 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92bc100be0 0x7f92bc195980 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f92a806c610 0x7f92a806eac0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 --2- 172.21.15.153:0/1675821207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92bc103500 0x7f92bc195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 >> 172.21.15.153:0/1675821207 conn(0x7f92bc0fa600 msgr2=0x7f92bc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.635+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 shutdown_connections 2024-05-06T14:25:40.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:40.636+0000 7f92c1da2700 1 -- 172.21.15.153:0/1675821207 wait complete. 2024-05-06T14:25:40.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:40.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:40.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:41.231 DEBUG:teuthology.orchestra.run.smithi153:osd.0> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.0.service 2024-05-06T14:25:41.234 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_3... 2024-05-06T14:25:41.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-06T14:25:41.708 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:25:41.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:41 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:41.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:41 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:41.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:41 smithi153 ceph-mon[27177]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:41.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:41 smithi153 ceph-mon[27177]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T14:25:41.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:25:41 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0[42069]: 2024-05-06T14:25:41.536+0000 7fc6b1ebb640 -1 osd.0 0 log_to_monitors true 2024-05-06T14:25:41.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:41 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:41.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:41 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:41.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:41 smithi190 ceph-mon[33671]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:41.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:41 smithi190 ceph-mon[33671]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T14:25:42.883 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:42 smithi153 ceph-mon[27177]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T14:25:42.883 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:42 smithi153 ceph-mon[27177]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T14:25:42.883 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:42.883 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:42 smithi153 ceph-mon[27177]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-05-06T14:25:42.884 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:25:42 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0[42069]: 2024-05-06T14:25:42.643+0000 7fc6a6d31700 -1 osd.0 0 waiting for initial osdmap 2024-05-06T14:25:42.884 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:25:42 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0[42069]: 2024-05-06T14:25:42.646+0000 7fc6a2325700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:25:42.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:42 smithi190 ceph-mon[33671]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T14:25:42.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:42 smithi190 ceph-mon[33671]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T14:25:42.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:42.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:42 smithi190 ceph-mon[33671]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128961 s, 813 MB/s 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 2024-05-06T14:25:43.717 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-05-06T14:25:43.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:43 smithi190 ceph-mon[33671]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-05-06T14:25:43.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:43 smithi190 ceph-mon[33671]: osdmap e8: 1 total, 0 up, 1 in 2024-05-06T14:25:43.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:43 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:43.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:43 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:43.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:43 smithi190 ceph-mon[33671]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:44.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:43 smithi153 ceph-mon[27177]: from='osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-05-06T14:25:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:43 smithi153 ceph-mon[27177]: osdmap e8: 1 total, 0 up, 1 in 2024-05-06T14:25:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:43 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:43 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:43 smithi153 ceph-mon[27177]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T14:25:44.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:44 smithi153 ceph-mon[27177]: purged_snaps scrub starts 2024-05-06T14:25:44.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:44 smithi153 ceph-mon[27177]: purged_snaps scrub ok 2024-05-06T14:25:44.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:44 smithi153 ceph-mon[27177]: osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] boot 2024-05-06T14:25:44.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:44 smithi153 ceph-mon[27177]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T14:25:44.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:44 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:44.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:44 smithi190 ceph-mon[33671]: purged_snaps scrub starts 2024-05-06T14:25:44.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:44 smithi190 ceph-mon[33671]: purged_snaps scrub ok 2024-05-06T14:25:44.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:44 smithi190 ceph-mon[33671]: osd.0 [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] boot 2024-05-06T14:25:44.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:44 smithi190 ceph-mon[33671]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T14:25:44.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:44 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:25:45.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-05-06T14:25:45.392 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:25:46.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:45 smithi153 ceph-mon[27177]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:46.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:45 smithi153 ceph-mon[27177]: osdmap e10: 1 total, 1 up, 1 in 2024-05-06T14:25:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:45 smithi190 ceph-mon[33671]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:45 smithi190 ceph-mon[33671]: osdmap e10: 1 total, 1 up, 1 in 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.388+0000 7ff227162700 1 -- 172.21.15.153:0/2275705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 msgr2=0x7ff220100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.388+0000 7ff227162700 1 --2- 172.21.15.153:0/2275705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff220100a60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff214009910 tx=0x7ff214009c20 comp rx=0 tx=0).stop 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.388+0000 7ff227162700 1 -- 172.21.15.153:0/2275705322 shutdown_connections 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.388+0000 7ff227162700 1 --2- 172.21.15.153:0/2275705322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 0x7ff2200ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.388+0000 7ff227162700 1 --2- 172.21.15.153:0/2275705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff220100a60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.388+0000 7ff227162700 1 -- 172.21.15.153:0/2275705322 >> 172.21.15.153:0/2275705322 conn(0x7ff2200f9dd0 msgr2=0x7ff2200fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:25:47.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 -- 172.21.15.153:0/2275705322 shutdown_connections 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 -- 172.21.15.153:0/2275705322 wait complete. 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 Processor -- start 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 -- start start 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 0x7ff2200fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff22019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22019ad90 con 0x7ff220100650 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff227162700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22019af00 con 0x7ff2200ff390 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff21ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff22019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff21ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff22019a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54524/0 (socket says 172.21.15.153:54524) 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.389+0000 7ff21ffff700 1 -- 172.21.15.153:0/3782845237 learned_addr learned my addr 172.21.15.153:0/3782845237 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff224efe700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 0x7ff2200fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21ffff700 1 -- 172.21.15.153:0/3782845237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 msgr2=0x7ff2200fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:25:47.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21ffff700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 0x7ff2200fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21ffff700 1 -- 172.21.15.153:0/3782845237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140095f0 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff224efe700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 0x7ff2200fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21ffff700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff22019a7c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00b4b0 tx=0x7ff20c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20c011800 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff20c011e10 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22019b1e0 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20c00f2e0 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.390+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22019b7b0 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.391+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff220107c80 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.392+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7ff20c010380 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.392+0000 7ff21dffb700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff21006c600 0x7ff21006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.392+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7ff20c08a520 con 0x7ff220100650 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.392+0000 7ff224efe700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff21006c600 0x7ff21006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:25:47.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.392+0000 7ff224efe700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff21006c600 0x7ff21006eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b5c0 tx=0x7ff214005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:25:47.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.396+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff20c059640 con 0x7ff220100650 2024-05-06T14:25:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:25:47.608+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff220104cf0 con 0x7ff21006c600 2024-05-06T14:25:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:47 smithi153 ceph-mon[27177]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:47 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:25:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:47 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:25:48.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:47 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:48.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:47 smithi190 ceph-mon[33671]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:48.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:47 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:25:48.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:47 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:25:48.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:47 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:49.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:49 smithi153 ceph-mon[27177]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:25:49.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:49 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:49.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:49 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:49.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:49 smithi190 ceph-mon[33671]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:25:49.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:49 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:49.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:49 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:50.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:50 smithi153 ceph-mon[27177]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:50.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:50 smithi190 ceph-mon[33671]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:51.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:51 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3731140101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24a1c7ac-a168-4e7d-8c34-3e4c73408756"}]: dispatch 2024-05-06T14:25:51.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:51 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3731140101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24a1c7ac-a168-4e7d-8c34-3e4c73408756"}]': finished 2024-05-06T14:25:51.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:51 smithi190 ceph-mon[33671]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T14:25:51.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:51 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:25:51.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:51 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2796624622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:25:51.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:51 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3731140101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24a1c7ac-a168-4e7d-8c34-3e4c73408756"}]: dispatch 2024-05-06T14:25:51.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:51 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3731140101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24a1c7ac-a168-4e7d-8c34-3e4c73408756"}]': finished 2024-05-06T14:25:51.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:51 smithi153 ceph-mon[27177]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T14:25:51.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:51 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:25:51.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:51 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2796624622' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:25:52.587 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:52 smithi190 ceph-mon[33671]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:52.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:52 smithi153 ceph-mon[27177]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:54.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:54 smithi153 ceph-mon[27177]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:54.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:54 smithi190 ceph-mon[33671]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:55.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:25:56.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:56 smithi153 ceph-mon[27177]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:56.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:56 smithi190 ceph-mon[33671]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:58.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:58.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: Detected new or changed devices on smithi153 2024-05-06T14:25:58.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:25:58.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi153 to 11840M 2024-05-06T14:25:58.033 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.033 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:58.033 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:25:58.033 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.033 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:25:58.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:25:58.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: Detected new or changed devices on smithi153 2024-05-06T14:25:58.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:25:58.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi153 to 11840M 2024-05-06T14:25:58.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:25:58.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:25:58.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:25:58.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:00.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:59 smithi153 ceph-mon[27177]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T14:26:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:59 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:25:59 smithi153 ceph-mon[27177]: Deploying daemon osd.1 on smithi153 2024-05-06T14:26:00.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:00.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:00.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:59 smithi190 ceph-mon[33671]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:00.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T14:26:00.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:59 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:00.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:25:59 smithi190 ceph-mon[33671]: Deploying daemon osd.1 on smithi153 2024-05-06T14:26:02.244 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:01 smithi153 ceph-mon[27177]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:02.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:01 smithi190 ceph-mon[33671]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:04 smithi190 ceph-mon[33671]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:04 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:04.205 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:03 smithi153 ceph-mon[27177]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:04.205 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:04.205 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:04.205 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:04.205 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:04.205 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:04 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:06.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:06 smithi153 ceph-mon[27177]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:06.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:06 smithi190 ceph-mon[33671]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:07.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:07.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:07.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:07.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:07 smithi190 ceph-mon[33671]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:07 smithi153 ceph-mon[27177]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:09.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:09 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:09.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:09 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:09.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:09 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:09.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:09 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:09.870 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-05-06T14:26:09.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.882+0000 7ff21dffb700 1 -- 172.21.15.153:0/3782845237 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff220104cf0 con 0x7ff21006c600 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff21006c600 msgr2=0x7ff21006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff21006c600 0x7ff21006eab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b5c0 tx=0x7ff214005960 comp rx=0 tx=0).stop 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 msgr2=0x7ff22019a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff22019a7c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00b4b0 tx=0x7ff20c00b870 comp rx=0 tx=0).stop 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 shutdown_connections 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff21006c600 0x7ff21006eab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2200ff390 0x7ff2200fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 --2- 172.21.15.153:0/3782845237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff220100650 0x7ff22019a7c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 >> 172.21.15.153:0/3782845237 conn(0x7ff2200f9dd0 msgr2=0x7ff2201035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 shutdown_connections 2024-05-06T14:26:09.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:09.885+0000 7ff227162700 1 -- 172.21.15.153:0/3782845237 wait complete. 2024-05-06T14:26:10.481 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:10 smithi153 ceph-mon[27177]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:10.481 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:10.481 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:10.481 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:10.481 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:10 smithi153 ceph-mon[27177]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T14:26:10.502 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.1.service 2024-05-06T14:26:10.505 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_2... 2024-05-06T14:26:10.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-06T14:26:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:10 smithi190 ceph-mon[33671]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:10 smithi190 ceph-mon[33671]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T14:26:10.938 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:26:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:11 smithi153 ceph-mon[27177]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T14:26:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:11 smithi153 ceph-mon[27177]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T14:26:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:11 smithi153 ceph-mon[27177]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-05-06T14:26:12.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:11 smithi153 ceph-mon[27177]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:12.033 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:11 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1[47617]: 2024-05-06T14:26:11.872+0000 7fdf54943700 -1 osd.1 0 waiting for initial osdmap 2024-05-06T14:26:12.033 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:11 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1[47617]: 2024-05-06T14:26:11.877+0000 7fdf50738700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:26:12.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:11 smithi190 ceph-mon[33671]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T14:26:12.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:11 smithi190 ceph-mon[33671]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T14:26:12.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:12.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:11 smithi190 ceph-mon[33671]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-05-06T14:26:12.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:11 smithi190 ceph-mon[33671]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-06T14:26:13.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:12 smithi153 ceph-mon[27177]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-05-06T14:26:13.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:12 smithi153 ceph-mon[27177]: osdmap e13: 2 total, 1 up, 2 in 2024-05-06T14:26:13.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:12 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:13.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:12 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:13.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:12 smithi190 ceph-mon[33671]: from='osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-05-06T14:26:13.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:12 smithi190 ceph-mon[33671]: osdmap e13: 2 total, 1 up, 2 in 2024-05-06T14:26:13.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:12 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:13.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:12 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-06T14:26:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-06T14:26:13.261 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-05-06T14:26:13.261 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-05-06T14:26:13.261 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134318 s, 781 MB/s 2024-05-06T14:26:13.262 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-05-06T14:26:14.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:13 smithi153 ceph-mon[27177]: purged_snaps scrub starts 2024-05-06T14:26:14.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:13 smithi153 ceph-mon[27177]: purged_snaps scrub ok 2024-05-06T14:26:14.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:13 smithi153 ceph-mon[27177]: osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] boot 2024-05-06T14:26:14.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:13 smithi153 ceph-mon[27177]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T14:26:14.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:13 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:14.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:13 smithi153 ceph-mon[27177]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:14.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-05-06T14:26:14.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:13 smithi190 ceph-mon[33671]: purged_snaps scrub starts 2024-05-06T14:26:14.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:13 smithi190 ceph-mon[33671]: purged_snaps scrub ok 2024-05-06T14:26:14.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:13 smithi190 ceph-mon[33671]: osd.1 [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] boot 2024-05-06T14:26:14.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:13 smithi190 ceph-mon[33671]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T14:26:14.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:13 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:26:14.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:13 smithi190 ceph-mon[33671]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:14.833 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:26:15.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:14 smithi153 ceph-mon[27177]: osdmap e15: 2 total, 2 up, 2 in 2024-05-06T14:26:15.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:14 smithi190 ceph-mon[33671]: osdmap e15: 2 total, 2 up, 2 in 2024-05-06T14:26:15.903 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:15 smithi153 ceph-mon[27177]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:15.903 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:15.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:16.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:15 smithi190 ceph-mon[33671]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:16.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:16.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:16.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.406+0000 7f2b63fe1700 1 -- 172.21.15.153:0/138415090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 msgr2=0x7f2b5c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:16.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.406+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/138415090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c1006a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009910 tx=0x7f2b4c009c20 comp rx=0 tx=0).stop 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- 172.21.15.153:0/138415090 shutdown_connections 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/138415090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c1006a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/138415090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b5c0ff050 0x7f2b5c0ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- 172.21.15.153:0/138415090 >> 172.21.15.153:0/138415090 conn(0x7f2b5c0fa5e0 msgr2=0x7f2b5c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- 172.21.15.153:0/138415090 shutdown_connections 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- 172.21.15.153:0/138415090 wait complete. 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 Processor -- start 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- start start 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b5c0ff050 0x7f2b5c102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c101b70 con 0x7f2b5c100250 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.407+0000 7f2b63fe1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c101cb0 con 0x7f2b5c0ff050 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:26:16.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34036/0 (socket says 172.21.15.153:34036) 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 -- 172.21.15.153:0/257808213 learned_addr learned my addr 172.21.15.153:0/257808213 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b61d7d700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b5c0ff050 0x7f2b5c102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 -- 172.21.15.153:0/257808213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b5c0ff050 msgr2=0x7f2b5c102fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b5c0ff050 0x7f2b5c102fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 -- 172.21.15.153:0/257808213 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0095f0 con 0x7f2b5c100250 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b6157c700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c101630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006010 tx=0x7f2b4c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b4c01d070 con 0x7f2b5c100250 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b4c00f460 con 0x7f2b5c100250 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b4c021640 con 0x7f2b5c100250 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c101f30 con 0x7f2b5c100250 2024-05-06T14:26:16.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.408+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c102380 con 0x7f2b5c100250 2024-05-06T14:26:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.409+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c107ba0 con 0x7f2b5c100250 2024-05-06T14:26:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.410+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f2b4c00fab0 con 0x7f2b5c100250 2024-05-06T14:26:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.410+0000 7f2b52ffd700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2b4806c600 0x7f2b4806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:26:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.410+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f2b4c08c060 con 0x7f2b5c100250 2024-05-06T14:26:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.410+0000 7f2b61d7d700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2b4806c600 0x7f2b4806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:26:16.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.410+0000 7f2b61d7d700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2b4806c600 0x7f2b4806eab0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b58005fd0 tx=0x7f2b58005dd0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:26:16.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.414+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2b4c05aea0 con 0x7f2b5c100250 2024-05-06T14:26:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:16.622+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b5c19de70 con 0x7f2b4806c600 2024-05-06T14:26:17.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:16 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:26:17.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:16 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:26:17.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:16 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:17.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:16 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:26:17.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:16 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:26:17.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:16 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:17 smithi153 ceph-mon[27177]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:26:17.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:17 smithi153 ceph-mon[27177]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:18.190 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:17 smithi190 ceph-mon[33671]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:26:18.190 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:17 smithi190 ceph-mon[33671]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:20.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:20.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi153 to 5920M 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:20 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:20.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:20.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:20.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:20.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi153 to 5920M 2024-05-06T14:26:20.337 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.337 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:20.337 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:20.337 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:20.337 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:20 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:21.340 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:21 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1726017591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b89edb2-1c0e-4481-91ed-157d767c0c05"}]: dispatch 2024-05-06T14:26:21.340 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:21 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1726017591' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b89edb2-1c0e-4481-91ed-157d767c0c05"}]': finished 2024-05-06T14:26:21.340 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:21 smithi153 ceph-mon[27177]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T14:26:21.340 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:21.340 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:21.341 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:21.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:21 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1726017591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b89edb2-1c0e-4481-91ed-157d767c0c05"}]: dispatch 2024-05-06T14:26:21.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:21 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1726017591' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b89edb2-1c0e-4481-91ed-157d767c0c05"}]': finished 2024-05-06T14:26:21.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:21 smithi190 ceph-mon[33671]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T14:26:21.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:21.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:21.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:22.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:22 smithi153 ceph-mon[27177]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:22.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:22 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/4240377458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:26:22.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:22 smithi190 ceph-mon[33671]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:22.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:22 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/4240377458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:26:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:24 smithi190 ceph-mon[33671]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:24.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:24.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:24 smithi153 ceph-mon[27177]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:24.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:24.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:24.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:24.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:24.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:25.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:25.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:26.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:26 smithi153 ceph-mon[27177]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:26.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:26 smithi190 ceph-mon[33671]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:28.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:28 smithi153 ceph-mon[27177]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:28.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T14:26:28.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:28 smithi190 ceph-mon[33671]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T14:26:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:29 smithi190 ceph-mon[33671]: Deploying daemon osd.2 on smithi153 2024-05-06T14:26:29.444 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:29 smithi153 ceph-mon[27177]: Deploying daemon osd.2 on smithi153 2024-05-06T14:26:30.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:30 smithi190 ceph-mon[33671]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:30.444 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:30 smithi153 ceph-mon[27177]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:32.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:32 smithi153 ceph-mon[27177]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:32.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:32 smithi190 ceph-mon[33671]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:34.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:34.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:33 smithi153 ceph-mon[27177]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:34.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:34.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:34.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:34.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:33 smithi190 ceph-mon[33671]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:34.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:34.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:36.239 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:35 smithi153 ceph-mon[27177]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:36.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.329+0000 7f2b52ffd700 1 -- 172.21.15.153:0/257808213 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2b5c19de70 con 0x7f2b4806c600 2024-05-06T14:26:36.319 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-05-06T14:26:36.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.331+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2b4806c600 msgr2=0x7f2b4806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.331+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2b4806c600 0x7f2b4806eab0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b58005fd0 tx=0x7f2b58005dd0 comp rx=0 tx=0).stop 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.331+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 msgr2=0x7f2b5c101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.331+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c101630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006010 tx=0x7f2b4c00be20 comp rx=0 tx=0).stop 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 shutdown_connections 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2b4806c600 0x7f2b4806eab0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b5c0ff050 0x7f2b5c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 --2- 172.21.15.153:0/257808213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b5c100250 0x7f2b5c101630 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 >> 172.21.15.153:0/257808213 conn(0x7f2b5c0fa5e0 msgr2=0x7f2b5c0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 shutdown_connections 2024-05-06T14:26:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:26:36.332+0000 7f2b63fe1700 1 -- 172.21.15.153:0/257808213 wait complete. 2024-05-06T14:26:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:35 smithi190 ceph-mon[33671]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:36.935 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.2.service 2024-05-06T14:26:36.938 INFO:tasks.cephadm:Deploying osd.3 on smithi190 with /dev/vg_nvme/lv_4... 2024-05-06T14:26:36.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-06T14:26:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:26:37.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:37.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:37 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:37.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:37 smithi153 ceph-mon[27177]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:26:37.574 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:37.574 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:37 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:37.574 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:37 smithi190 ceph-mon[33671]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:26:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:38 smithi190 ceph-mon[33671]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:38 smithi190 ceph-mon[33671]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T14:26:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:38 smithi190 ceph-mon[33671]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T14:26:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:38 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:38 smithi190 ceph-mon[33671]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-05-06T14:26:38.586 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:38 smithi153 ceph-mon[27177]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:38.586 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:38 smithi153 ceph-mon[27177]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T14:26:38.587 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:38 smithi153 ceph-mon[27177]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T14:26:38.587 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:38 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:38.587 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:38 smithi153 ceph-mon[27177]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-05-06T14:26:38.587 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:38 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2[53091]: 2024-05-06T14:26:38.298+0000 7f51e1f11700 -1 osd.2 0 waiting for initial osdmap 2024-05-06T14:26:38.587 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:38 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2[53091]: 2024-05-06T14:26:38.303+0000 7f51da4ff700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:26:38.979 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-06T14:26:38.979 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-06T14:26:38.979 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-05-06T14:26:38.979 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-05-06T14:26:38.979 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133134 s, 788 MB/s 2024-05-06T14:26:38.980 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-05-06T14:26:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:39 smithi153 ceph-mon[27177]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-05-06T14:26:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:39 smithi153 ceph-mon[27177]: osdmap e18: 3 total, 2 up, 3 in 2024-05-06T14:26:39.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:39.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:39 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:39.600 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:39 smithi190 ceph-mon[33671]: from='osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-05-06T14:26:39.600 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:39 smithi190 ceph-mon[33671]: osdmap e18: 3 total, 2 up, 3 in 2024-05-06T14:26:39.600 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:39.600 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:39 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:39.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-05-06T14:26:39.831 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: purged_snaps scrub starts 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: purged_snaps scrub ok 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] boot 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:40.506 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:40.507 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:40.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: purged_snaps scrub starts 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: purged_snaps scrub ok 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: osd.2 [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] boot 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:40.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 -- 172.21.15.190:0/1311602618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 msgr2=0x7f39b01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 --2- 172.21.15.190:0/1311602618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b01006a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3998009910 tx=0x7f3998009c20 comp rx=0 tx=0).stop 2024-05-06T14:26:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 -- 172.21.15.190:0/1311602618 shutdown_connections 2024-05-06T14:26:41.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 --2- 172.21.15.190:0/1311602618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b01006a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:41.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 --2- 172.21.15.190:0/1311602618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00ff050 0x7f39b00ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:41.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 -- 172.21.15.190:0/1311602618 >> 172.21.15.190:0/1311602618 conn(0x7f39b00fa600 msgr2=0x7f39b00fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:26:41.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 -- 172.21.15.190:0/1311602618 shutdown_connections 2024-05-06T14:26:41.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.350+0000 7f39b505e700 1 -- 172.21.15.190:0/1311602618 wait complete. 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39b505e700 1 Processor -- start 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39b505e700 1 -- start start 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39b505e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00ff050 0x7f39b0194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39b505e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b0194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39b505e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b01953c0 con 0x7f39b00ff050 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39b505e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0195500 con 0x7f39b0100250 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39a659c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b0194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39a659c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b0194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:50256/0 (socket says 172.21.15.190:50256) 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39aed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00ff050 0x7f39b0194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:26:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39a659c700 1 -- 172.21.15.190:0/2694079326 learned_addr learned my addr 172.21.15.190:0/2694079326 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39a659c700 1 -- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00ff050 msgr2=0x7f39b0194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.351+0000 7f39a659c700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00ff050 0x7f39b0194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39a659c700 1 -- 172.21.15.190:0/2694079326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980095f0 con 0x7f39b0100250 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39a659c700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b0194da0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3998000c00 tx=0x7f399800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f399801d070 con 0x7f39b0100250 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3998005dc0 con 0x7f39b0100250 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0199f50 con 0x7f39b0100250 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39980226a0 con 0x7f39b0100250 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.352+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b019a410 con 0x7f39b0100250 2024-05-06T14:26:41.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.353+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b0107ba0 con 0x7f39b0100250 2024-05-06T14:26:41.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.353+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 18) v1 ==== 89860+0+0 (secure 0 0 0) 0x7f39980038f0 con 0x7f39b0100250 2024-05-06T14:26:41.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.354+0000 7f39acd99700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f399c06c610 0x7f399c06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:26:41.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.354+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7f399808c270 con 0x7f39b0100250 2024-05-06T14:26:41.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.354+0000 7f39aed9d700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f399c06c610 0x7f399c06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:26:41.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.354+0000 7f39aed9d700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f399c06c610 0x7f399c06eac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f39b01000b0 tx=0x7f39a0009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:26:41.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.358+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f399805af00 con 0x7f39b0100250 2024-05-06T14:26:41.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:41.556+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f39b0104ba0 con 0x7f399c06c610 2024-05-06T14:26:41.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:41 smithi190 ceph-mon[33671]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T14:26:41.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:41 smithi190 ceph-mon[33671]: pgmap v76: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:41.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:41 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-06T14:26:41.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:41 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:26:41.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:41 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:26:41.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:41 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:41.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:41 smithi153 ceph-mon[27177]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T14:26:41.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:41 smithi153 ceph-mon[27177]: pgmap v76: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:41.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:41 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-06T14:26:41.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:41 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:26:41.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:41 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:26:41.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:41 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:43.001 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:26:43.001 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:43.002 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:43.003 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.003 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:43.003 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:43.003 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:43.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:43.011 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:43.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55540]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T14:26:43.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55540]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:43.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55540]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:43.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55540]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:44.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55547]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T14:26:44.032 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55547]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:44.032 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55547]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:44.032 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:26:43 smithi153 sudo[55547]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:43 smithi153 ceph-mon[27177]: Detected new or changed devices on smithi153 2024-05-06T14:26:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:43 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi153 to 3946M 2024-05-06T14:26:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:43 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T14:26:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:43 smithi153 ceph-mon[27177]: osdmap e22: 3 total, 3 up, 3 in 2024-05-06T14:26:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:43 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:44.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:43 smithi153 ceph-mon[27177]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:44.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:43 smithi190 ceph-mon[33671]: Detected new or changed devices on smithi153 2024-05-06T14:26:44.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:43 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi153 to 3946M 2024-05-06T14:26:44.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:43 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T14:26:44.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:43 smithi190 ceph-mon[33671]: osdmap e22: 3 total, 3 up, 3 in 2024-05-06T14:26:44.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:43 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:44.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:43 smithi190 ceph-mon[33671]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:44.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55646]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T14:26:44.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:44.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55646]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:44.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55646]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55670]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55670]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55670]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55673]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:44.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55673]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:44.695 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55673]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:44.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: osdmap e23: 3 total, 3 up, 3 in 2024-05-06T14:26:44.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:44.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1875373914' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74b7823e-2b78-4570-9b7a-89abc9d7fbb5"}]: dispatch 2024-05-06T14:26:44.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74b7823e-2b78-4570-9b7a-89abc9d7fbb5"}]: dispatch 2024-05-06T14:26:44.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74b7823e-2b78-4570-9b7a-89abc9d7fbb5"}]': finished 2024-05-06T14:26:44.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T14:26:44.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:26:44.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:44 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/574549526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: osdmap e23: 3 total, 3 up, 3 in 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/1875373914' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74b7823e-2b78-4570-9b7a-89abc9d7fbb5"}]: dispatch 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74b7823e-2b78-4570-9b7a-89abc9d7fbb5"}]: dispatch 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74b7823e-2b78-4570-9b7a-89abc9d7fbb5"}]': finished 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:26:45.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:44 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/574549526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:26:45.194 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55752]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T14:26:45.195 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55722]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T14:26:45.195 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55722]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:45.195 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55722]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:45.195 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:26:44 smithi153 sudo[55722]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:45.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55752]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:45.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55752]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:45.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55752]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:45.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55755]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T14:26:45.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55755]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:45.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55755]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:45.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 sudo[55755]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:45.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37978]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T14:26:45.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37978]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:45.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37978]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:45.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37978]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:45.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:45.974+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f3998057840 con 0x7f39b0100250 2024-05-06T14:26:46.116 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: pgmap v82: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:46.116 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T14:26:46.116 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T14:26:46.116 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:26:46.116 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:26:46.117 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:26:46.117 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:45 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37982]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37982]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37982]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 sudo[37982]: pam_unix(sudo:session): session closed for user root 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: pgmap v82: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T14:26:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:26:46.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:26:46.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:26:46.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:45 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:26:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:46 smithi153 ceph-mon[27177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T14:26:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:46 smithi153 ceph-mon[27177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T14:26:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:46 smithi153 ceph-mon[27177]: mgrmap e19: smithi153.dolfed(active, since 2m), standbys: smithi190.qauohf 2024-05-06T14:26:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:46 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:47.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:46 smithi190 ceph-mon[33671]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T14:26:47.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:46 smithi190 ceph-mon[33671]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T14:26:47.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:46 smithi190 ceph-mon[33671]: mgrmap e19: smithi153.dolfed(active, since 2m), standbys: smithi190.qauohf 2024-05-06T14:26:47.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:46 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:48.159 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:48 smithi190 ceph-mon[33671]: pgmap v83: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:48.159 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:48.159 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:48.159 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:48.159 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:48.160 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:48.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:48 smithi153 ceph-mon[27177]: pgmap v83: 1 pgs: 1 creating+activating; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:48.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:26:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:50.091 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:50 smithi190 ceph-mon[33671]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:50.491 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:50 smithi153 ceph-mon[27177]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:52 smithi190 ceph-mon[33671]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:52 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T14:26:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:52 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:52 smithi190 ceph-mon[33671]: Deploying daemon osd.3 on smithi190 2024-05-06T14:26:52.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:52 smithi153 ceph-mon[27177]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:52.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:52 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T14:26:52.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:52 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:26:52.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:52 smithi153 ceph-mon[27177]: Deploying daemon osd.3 on smithi190 2024-05-06T14:26:54.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:54 smithi190 ceph-mon[33671]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:54.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:54 smithi153 ceph-mon[27177]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:55.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:55.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:26:56.410 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:56 smithi190 ceph-mon[33671]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:56.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:56 smithi153 ceph-mon[27177]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:57.348 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:57.348 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:57.348 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:57 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:57.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:26:57.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:57.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:57 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:26:58.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:26:58 smithi153 ceph-mon[27177]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:58.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:26:58 smithi190 ceph-mon[33671]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:26:59.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.463+0000 7f39acd99700 1 -- 172.21.15.190:0/2694079326 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f39b0104ba0 con 0x7f399c06c610 2024-05-06T14:26:59.453 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 3 on host 'smithi190' 2024-05-06T14:26:59.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f399c06c610 msgr2=0x7f399c06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:59.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f399c06c610 0x7f399c06eac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f39b01000b0 tx=0x7f39a0009260 comp rx=0 tx=0).stop 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 msgr2=0x7f39b0194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b0194da0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3998000c00 tx=0x7f399800f690 comp rx=0 tx=0).stop 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 shutdown_connections 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00ff050 0x7f39b0194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f399c06c610 0x7f399c06eac0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 --2- 172.21.15.190:0/2694079326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39b0100250 0x7f39b0194da0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 >> 172.21.15.190:0/2694079326 conn(0x7f39b00fa600 msgr2=0x7f39b0103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 shutdown_connections 2024-05-06T14:26:59.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:26:59.466+0000 7f39b505e700 1 -- 172.21.15.190:0/2694079326 wait complete. 2024-05-06T14:27:00.397 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:00 smithi190 ceph-mon[33671]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:27:00.397 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:00 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:00.397 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:00 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:00.397 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:00 smithi190 ceph-mon[33671]: from='osd.3 [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:27:00.397 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:00 smithi190 ceph-mon[33671]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:27:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:00 smithi153 ceph-mon[27177]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:27:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:00 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:00 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:00.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:00 smithi153 ceph-mon[27177]: from='osd.3 [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:27:00.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:00 smithi153 ceph-mon[27177]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:27:00.643 DEBUG:teuthology.orchestra.run.smithi190:osd.3> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.3.service 2024-05-06T14:27:00.646 INFO:tasks.cephadm:Deploying osd.4 on smithi190 with /dev/vg_nvme/lv_3... 2024-05-06T14:27:00.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-06T14:27:00.893 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:27:01.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T14:27:01.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: osdmap e25: 4 total, 3 up, 4 in 2024-05-06T14:27:01.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:01.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: from='osd.3 [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:01.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:01.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:01.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:01.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:01 smithi190 ceph-mon[33671]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:27:01.687 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:27:01 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:27:01.441+0000 7fd3e2dea700 -1 osd.3 0 waiting for initial osdmap 2024-05-06T14:27:01.687 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:27:01 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:27:01.447+0000 7fd3df3e0700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:27:01.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T14:27:01.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: osdmap e25: 4 total, 3 up, 4 in 2024-05-06T14:27:01.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:01.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: from='osd.3 [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:01.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:01.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:01.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:01.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:01 smithi153 ceph-mon[27177]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: osdmap e26: 4 total, 3 up, 4 in 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: osd.3 [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] boot 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: osdmap e27: 4 total, 4 up, 4 in 2024-05-06T14:27:02.676 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:02 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:02.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-05-06T14:27:02.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: osdmap e26: 4 total, 3 up, 4 in 2024-05-06T14:27:02.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:02.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:02.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: osd.3 [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] boot 2024-05-06T14:27:02.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: osdmap e27: 4 total, 4 up, 4 in 2024-05-06T14:27:02.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:02 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:27:03.245 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-06T14:27:03.245 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-06T14:27:03.245 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-05-06T14:27:03.245 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-05-06T14:27:03.246 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123018 s, 852 MB/s 2024-05-06T14:27:03.246 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-05-06T14:27:03.633 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:03 smithi190 ceph-mon[33671]: purged_snaps scrub starts 2024-05-06T14:27:03.634 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:03 smithi190 ceph-mon[33671]: purged_snaps scrub ok 2024-05-06T14:27:03.634 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:03 smithi190 ceph-mon[33671]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:03.634 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:03 smithi190 ceph-mon[33671]: osdmap e28: 4 total, 4 up, 4 in 2024-05-06T14:27:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:03 smithi153 ceph-mon[27177]: purged_snaps scrub starts 2024-05-06T14:27:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:03 smithi153 ceph-mon[27177]: purged_snaps scrub ok 2024-05-06T14:27:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:03 smithi153 ceph-mon[27177]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:03.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:03 smithi153 ceph-mon[27177]: osdmap e28: 4 total, 4 up, 4 in 2024-05-06T14:27:03.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-05-06T14:27:04.372 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:27:05.646 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:05 smithi190 ceph-mon[33671]: osdmap e29: 4 total, 4 up, 4 in 2024-05-06T14:27:05.646 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:05 smithi190 ceph-mon[33671]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:05.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:05 smithi153 ceph-mon[27177]: osdmap e29: 4 total, 4 up, 4 in 2024-05-06T14:27:05.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:05 smithi153 ceph-mon[27177]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: osdmap e30: 4 total, 4 up, 4 in 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: Detected new or changed devices on smithi190 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi190 to 14912M 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.510 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:06.511 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:06.511 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.511 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:06.511 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:06.511 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:06.511 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:06 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.622+0000 7f6038dc2700 1 -- 172.21.15.190:0/2521862523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 msgr2=0x7f60341017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.622+0000 7f6038dc2700 1 --2- 172.21.15.190:0/2521862523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f60341017e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6028009910 tx=0x7f6028009c20 comp rx=0 tx=0).stop 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 -- 172.21.15.190:0/2521862523 shutdown_connections 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 --2- 172.21.15.190:0/2521862523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f60341017e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 --2- 172.21.15.190:0/2521862523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034100190 0x7f60341005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 -- 172.21.15.190:0/2521862523 >> 172.21.15.190:0/2521862523 conn(0x7f60340fb720 msgr2=0x7f60340fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 -- 172.21.15.190:0/2521862523 shutdown_connections 2024-05-06T14:27:06.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 -- 172.21.15.190:0/2521862523 wait complete. 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 Processor -- start 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.623+0000 7f6038dc2700 1 -- start start 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6038dc2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034100190 0x7f6034074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6038dc2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f6034073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6038dc2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6034073680 con 0x7f6034100190 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6038dc2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60340737c0 con 0x7f6034101390 2024-05-06T14:27:06.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f6034073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f6034073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:60566/0 (socket says 172.21.15.190:60566) 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 -- 172.21.15.190:0/4152711832 learned_addr learned my addr 172.21.15.190:0/4152711832 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f603259c700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034100190 0x7f6034074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 -- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034100190 msgr2=0x7f6034074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034100190 0x7f6034074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 -- 172.21.15.190:0/4152711832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280095f0 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6031d9b700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f6034073140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6028000c00 tx=0x7f602800f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f602801d070 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6028005dc0 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60280226a0 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6034073a40 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.624+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6034073f30 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.625+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6034108ce0 con 0x7f6034101390 2024-05-06T14:27:06.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.626+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f6028003680 con 0x7f6034101390 2024-05-06T14:27:06.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.626+0000 7f60237fe700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f601c06c680 0x7f601c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:06.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.626+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f602808c7d0 con 0x7f6034101390 2024-05-06T14:27:06.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.626+0000 7f603259c700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f601c06c680 0x7f601c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:06.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.627+0000 7f603259c700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f601c06c680 0x7f601c06eb30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f60341011f0 tx=0x7f6024009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:06.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.631+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f602805afe0 con 0x7f6034101390 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: osdmap e30: 4 total, 4 up, 4 in 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: Detected new or changed devices on smithi190 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi190 to 14912M 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:06.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:06.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:06.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:06 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:06.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:06.827+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6034105ce0 con 0x7f601c06c680 2024-05-06T14:27:07.587 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:07 smithi190 ceph-mon[33671]: osdmap e31: 4 total, 4 up, 4 in 2024-05-06T14:27:07.587 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:07 smithi190 ceph-mon[33671]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:27:07.588 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:27:07.588 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:27:07.588 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:07.588 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:07 smithi190 ceph-mon[33671]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:07.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:07 smithi153 ceph-mon[27177]: osdmap e31: 4 total, 4 up, 4 in 2024-05-06T14:27:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:07 smithi153 ceph-mon[27177]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:27:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:27:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:27:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:07.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:07 smithi153 ceph-mon[27177]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:10.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-05-06T14:27:10.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/2435692759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01290c44-3bf1-4da3-bac3-4dff2453cedb"}]: dispatch 2024-05-06T14:27:10.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01290c44-3bf1-4da3-bac3-4dff2453cedb"}]: dispatch 2024-05-06T14:27:10.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01290c44-3bf1-4da3-bac3-4dff2453cedb"}]': finished 2024-05-06T14:27:10.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: osdmap e32: 5 total, 4 up, 5 in 2024-05-06T14:27:10.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:10.187 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:10.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-05-06T14:27:10.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/2435692759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01290c44-3bf1-4da3-bac3-4dff2453cedb"}]: dispatch 2024-05-06T14:27:10.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01290c44-3bf1-4da3-bac3-4dff2453cedb"}]: dispatch 2024-05-06T14:27:10.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01290c44-3bf1-4da3-bac3-4dff2453cedb"}]': finished 2024-05-06T14:27:10.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: osdmap e32: 5 total, 4 up, 5 in 2024-05-06T14:27:10.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:10.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:11.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:11 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/1594957276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:27:11.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:11 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/1594957276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:27:12.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:12 smithi153 ceph-mon[27177]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-06T14:27:12.438 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:12 smithi190 ceph-mon[33671]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-06T14:27:14.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:14 smithi153 ceph-mon[27177]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-06T14:27:14.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:14 smithi190 ceph-mon[33671]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-05-06T14:27:16.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:16 smithi153 ceph-mon[27177]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-06T14:27:16.378 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:16 smithi190 ceph-mon[33671]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-06T14:27:17.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T14:27:17.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:17.305 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T14:27:17.305 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:18 smithi153 ceph-mon[27177]: Deploying daemon osd.4 on smithi190 2024-05-06T14:27:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:18 smithi153 ceph-mon[27177]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-06T14:27:18.308 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:18 smithi190 ceph-mon[33671]: Deploying daemon osd.4 on smithi190 2024-05-06T14:27:18.308 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:18 smithi190 ceph-mon[33671]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-06T14:27:20.261 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:20 smithi190 ceph-mon[33671]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:20.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:20 smithi153 ceph-mon[27177]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:22.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:22 smithi190 ceph-mon[33671]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:22.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:22 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:22.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:22 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:22.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:22 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:22.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:22 smithi153 ceph-mon[27177]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:22.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:22 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:22.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:22 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:22.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:22 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:24.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:24 smithi190 ceph-mon[33671]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:24.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:24 smithi153 ceph-mon[27177]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:25.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:25.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:25.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.840+0000 7f60237fe700 1 -- 172.21.15.190:0/4152711832 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6034105ce0 con 0x7f601c06c680 2024-05-06T14:27:25.827 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 4 on host 'smithi190' 2024-05-06T14:27:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f601c06c680 msgr2=0x7f601c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f601c06c680 0x7f601c06eb30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f60341011f0 tx=0x7f6024009260 comp rx=0 tx=0).stop 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 msgr2=0x7f6034073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f6034073140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6028000c00 tx=0x7f602800f740 comp rx=0 tx=0).stop 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 shutdown_connections 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6034100190 0x7f6034074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f601c06c680 0x7f601c06eb30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 --2- 172.21.15.190:0/4152711832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6034101390 0x7f6034073140 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.842+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 >> 172.21.15.190:0/4152711832 conn(0x7f60340fb720 msgr2=0x7f60341045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.843+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 shutdown_connections 2024-05-06T14:27:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:25.843+0000 7f6038dc2700 1 -- 172.21.15.190:0/4152711832 wait complete. 2024-05-06T14:27:26.395 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:26 smithi190 ceph-mon[33671]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:26.395 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:26 smithi190 ceph-mon[33671]: from='osd.4 [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:27:26.395 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:26 smithi190 ceph-mon[33671]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:27:26.395 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:26 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:26.396 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:26 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:26.416 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.4.service 2024-05-06T14:27:26.418 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_2... 2024-05-06T14:27:26.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-06T14:27:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:26 smithi153 ceph-mon[27177]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:26 smithi153 ceph-mon[27177]: from='osd.4 [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:27:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:26 smithi153 ceph-mon[27177]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:27:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:26 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:26 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:26.667 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: osdmap e33: 5 total, 4 up, 5 in 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: from='osd.4 [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:27.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:27 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:27.437 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:27:27 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:27:27.064+0000 7fd5e4afb700 -1 osd.4 0 waiting for initial osdmap 2024-05-06T14:27:27.437 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:27:27 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:27:27.071+0000 7fd5db8e6700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:27:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T14:27:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: osdmap e33: 5 total, 4 up, 5 in 2024-05-06T14:27:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: from='osd.4 [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:27.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:27.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:27.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:27 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:28.323 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:28 smithi190 ceph-mon[33671]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:28.323 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:28 smithi190 ceph-mon[33671]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-05-06T14:27:28.323 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:28 smithi190 ceph-mon[33671]: osdmap e34: 5 total, 4 up, 5 in 2024-05-06T14:27:28.323 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:28.323 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:28.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:28 smithi153 ceph-mon[27177]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:28.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:28 smithi153 ceph-mon[27177]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-05-06T14:27:28.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:28 smithi153 ceph-mon[27177]: osdmap e34: 5 total, 4 up, 5 in 2024-05-06T14:27:28.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:28.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:28.716 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-06T14:27:28.716 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-06T14:27:28.716 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-05-06T14:27:28.716 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-05-06T14:27:28.717 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139011 s, 754 MB/s 2024-05-06T14:27:28.717 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-05-06T14:27:29.370 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:29 smithi190 ceph-mon[33671]: purged_snaps scrub starts 2024-05-06T14:27:29.370 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:29 smithi190 ceph-mon[33671]: purged_snaps scrub ok 2024-05-06T14:27:29.370 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:29 smithi190 ceph-mon[33671]: osd.4 [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] boot 2024-05-06T14:27:29.370 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:29 smithi190 ceph-mon[33671]: osdmap e35: 5 total, 5 up, 5 in 2024-05-06T14:27:29.370 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:29 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:29 smithi153 ceph-mon[27177]: purged_snaps scrub starts 2024-05-06T14:27:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:29 smithi153 ceph-mon[27177]: purged_snaps scrub ok 2024-05-06T14:27:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:29 smithi153 ceph-mon[27177]: osd.4 [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] boot 2024-05-06T14:27:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:29 smithi153 ceph-mon[27177]: osdmap e35: 5 total, 5 up, 5 in 2024-05-06T14:27:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:29 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:27:29.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-05-06T14:27:30.344 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:30 smithi190 ceph-mon[33671]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:30.344 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:30 smithi190 ceph-mon[33671]: osdmap e36: 5 total, 5 up, 5 in 2024-05-06T14:27:30.379 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:27:30.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:30 smithi153 ceph-mon[27177]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-05-06T14:27:30.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:30 smithi153 ceph-mon[27177]: osdmap e36: 5 total, 5 up, 5 in 2024-05-06T14:27:32.114 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:32.114 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.114 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:32.115 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:32.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:32.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 -- 172.21.15.190:0/637479362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a340fe410 msgr2=0x7f4a340fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 --2- 172.21.15.190:0/637479362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a340fe410 0x7f4a340fe820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009910 tx=0x7f4a24009c20 comp rx=0 tx=0).stop 2024-05-06T14:27:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 -- 172.21.15.190:0/637479362 shutdown_connections 2024-05-06T14:27:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 --2- 172.21.15.190:0/637479362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a341013f0 0x7f4a34101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 --2- 172.21.15.190:0/637479362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a340fe410 0x7f4a340fe820 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 -- 172.21.15.190:0/637479362 >> 172.21.15.190:0/637479362 conn(0x7f4a340f9d80 msgr2=0x7f4a340fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 -- 172.21.15.190:0/637479362 shutdown_connections 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 -- 172.21.15.190:0/637479362 wait complete. 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.618+0000 7f4a3954e700 1 Processor -- start 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a3954e700 1 -- start start 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a3954e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a340fe410 0x7f4a34100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a3954e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 0x7f4a340ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a3954e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a340ff8e0 con 0x7f4a340fe410 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a3954e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a340ffa20 con 0x7f4a341013f0 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 0x7f4a340ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 0x7f4a340ff3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:55026/0 (socket says 172.21.15.190:55026) 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 -- 172.21.15.190:0/4163257005 learned_addr learned my addr 172.21.15.190:0/4163257005 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a32ffd700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a340fe410 0x7f4a34100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 -- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a340fe410 msgr2=0x7f4a34100d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a340fe410 0x7f4a34100d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 -- 172.21.15.190:0/4163257005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240095f0 con 0x7f4a341013f0 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.619+0000 7f4a2a7fc700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 0x7f4a340ff3a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00d8d0 tx=0x7f4a1c00dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.620+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a1c0098a0 con 0x7f4a341013f0 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.620+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a340ffd00 con 0x7f4a341013f0 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.620+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4a1c010460 con 0x7f4a341013f0 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.620+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a1c00f5f0 con 0x7f4a341013f0 2024-05-06T14:27:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.620+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3419d9c0 con 0x7f4a341013f0 2024-05-06T14:27:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.621+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a34108c90 con 0x7f4a341013f0 2024-05-06T14:27:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.621+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f4a1c009a00 con 0x7f4a341013f0 2024-05-06T14:27:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.622+0000 7f4a30ff9700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f4a2006c680 0x7f4a2006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.622+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f4a1c08b270 con 0x7f4a341013f0 2024-05-06T14:27:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.622+0000 7f4a32ffd700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f4a2006c680 0x7f4a2006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.622+0000 7f4a32ffd700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f4a2006c680 0x7f4a2006eb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a24006010 tx=0x7f4a2400b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:32.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.626+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4a1c0598d0 con 0x7f4a341013f0 2024-05-06T14:27:32.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:32.823+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a34105d40 con 0x7f4a2006c680 2024-05-06T14:27:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:33 smithi153 ceph-mon[27177]: Detected new or changed devices on smithi190 2024-05-06T14:27:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:33 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi190 to 7456M 2024-05-06T14:27:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:27:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:27:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:33 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:33.401 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:33 smithi190 ceph-mon[33671]: Detected new or changed devices on smithi190 2024-05-06T14:27:33.401 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:33 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi190 to 7456M 2024-05-06T14:27:33.401 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T14:27:33.401 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T14:27:33.401 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:33 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:34.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:34 smithi190 ceph-mon[33671]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:27:34.227 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:34 smithi190 ceph-mon[33671]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:34.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:34 smithi153 ceph-mon[27177]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:27:34.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:34 smithi153 ceph-mon[27177]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:36.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:36.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/4017479992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be366c2e-63d4-4496-b401-772dbe1a2444"}]: dispatch 2024-05-06T14:27:36.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be366c2e-63d4-4496-b401-772dbe1a2444"}]: dispatch 2024-05-06T14:27:36.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be366c2e-63d4-4496-b401-772dbe1a2444"}]': finished 2024-05-06T14:27:36.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T14:27:36.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:36.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:36 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/248727231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/4017479992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be366c2e-63d4-4496-b401-772dbe1a2444"}]: dispatch 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be366c2e-63d4-4496-b401-772dbe1a2444"}]: dispatch 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be366c2e-63d4-4496-b401-772dbe1a2444"}]': finished 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:36.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:36 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/248727231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T14:27:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:38 smithi190 ceph-mon[33671]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:38 smithi153 ceph-mon[27177]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:40.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:40 smithi190 ceph-mon[33671]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:40.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:40.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:40 smithi153 ceph-mon[27177]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:40.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:42.285 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:42 smithi190 ceph-mon[33671]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:42 smithi153 ceph-mon[27177]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:43.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:43 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T14:27:43.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:43 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:43.350 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:43 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T14:27:43.350 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:43 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:44.410 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:44 smithi190 ceph-mon[33671]: Deploying daemon osd.5 on smithi190 2024-05-06T14:27:44.410 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:44 smithi190 ceph-mon[33671]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:44.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:44 smithi153 ceph-mon[27177]: Deploying daemon osd.5 on smithi190 2024-05-06T14:27:44.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:44 smithi153 ceph-mon[27177]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:46.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:46 smithi153 ceph-mon[27177]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:46.322 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:46 smithi190 ceph-mon[33671]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:48.378 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:48 smithi190 ceph-mon[33671]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:48.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:48.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:48.379 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:48 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:48.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:48 smithi153 ceph-mon[27177]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:48.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:48 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:50.434 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:50 smithi190 ceph-mon[33671]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:50 smithi153 ceph-mon[27177]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:51.574 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.587+0000 7f4a30ff9700 1 -- 172.21.15.190:0/4163257005 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4a34105d40 con 0x7f4a2006c680 2024-05-06T14:27:51.574 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-05-06T14:27:51.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f4a2006c680 msgr2=0x7f4a2006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:51.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f4a2006c680 0x7f4a2006eb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a24006010 tx=0x7f4a2400b540 comp rx=0 tx=0).stop 2024-05-06T14:27:51.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 msgr2=0x7f4a340ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 0x7f4a340ff3a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00d8d0 tx=0x7f4a1c00dbe0 comp rx=0 tx=0).stop 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 shutdown_connections 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a340fe410 0x7f4a34100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f4a2006c680 0x7f4a2006eb30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 --2- 172.21.15.190:0/4163257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a341013f0 0x7f4a340ff3a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 >> 172.21.15.190:0/4163257005 conn(0x7f4a340f9d80 msgr2=0x7f4a34104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 shutdown_connections 2024-05-06T14:27:51.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:27:51.589+0000 7f4a3954e700 1 -- 172.21.15.190:0/4163257005 wait complete. 2024-05-06T14:27:52.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:52 smithi190 ceph-mon[33671]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:52 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:52 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:52 smithi190 ceph-mon[33671]: from='osd.5 [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:27:52.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:52 smithi190 ceph-mon[33671]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:27:52.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:52 smithi153 ceph-mon[27177]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:52.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:52 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:52.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:52 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:52.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:52 smithi153 ceph-mon[27177]: from='osd.5 [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:27:52.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:52 smithi153 ceph-mon[27177]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:27:52.576 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.5.service 2024-05-06T14:27:52.579 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-05-06T14:27:52.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-06T14:27:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:27:53.826 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:27:53 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:27:53.588+0000 7f157a5d2700 -1 osd.5 0 waiting for initial osdmap 2024-05-06T14:27:53.826 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:27:53 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:27:53.596+0000 7f15753c5700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: osdmap e38: 6 total, 5 up, 6 in 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: from='osd.5 [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:53.826 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:53.827 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:53 smithi190 ceph-mon[33671]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:53.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T14:27:53.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: osdmap e38: 6 total, 5 up, 6 in 2024-05-06T14:27:53.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:53.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: from='osd.5 [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:53.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-05-06T14:27:53.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:53.921 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:53.921 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:53 smithi153 ceph-mon[27177]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:54.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 -- 172.21.15.153:0/3938776766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 msgr2=0x7f6a540ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:54.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 --2- 172.21.15.153:0/3938776766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540ff470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009910 tx=0x7f6a48009c20 comp rx=0 tx=0).stop 2024-05-06T14:27:54.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 -- 172.21.15.153:0/3938776766 shutdown_connections 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 --2- 172.21.15.153:0/3938776766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a54100260 0x7f6a541006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 --2- 172.21.15.153:0/3938776766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540ff470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 -- 172.21.15.153:0/3938776766 >> 172.21.15.153:0/3938776766 conn(0x7f6a540fa610 msgr2=0x7f6a540fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 -- 172.21.15.153:0/3938776766 shutdown_connections 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.509+0000 7f6a5acab700 1 -- 172.21.15.153:0/3938776766 wait complete. 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a5acab700 1 Processor -- start 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a5acab700 1 -- start start 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a5acab700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a5acab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a54100260 0x7f6a5419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a5acab700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5419ac90 con 0x7f6a540ff060 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a5acab700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5419ae00 con 0x7f6a54100260 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a58a47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a58a47700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43406/0 (socket says 172.21.15.153:43406) 2024-05-06T14:27:54.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a58a47700 1 -- 172.21.15.153:0/655669085 learned_addr learned my addr 172.21.15.153:0/655669085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a53fff700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a54100260 0x7f6a5419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a58a47700 1 -- 172.21.15.153:0/655669085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a54100260 msgr2=0x7f6a5419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a58a47700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a54100260 0x7f6a5419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.510+0000 7f6a58a47700 1 -- 172.21.15.153:0/655669085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a480095f0 con 0x7f6a540ff060 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.511+0000 7f6a58a47700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540698f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800bda0 tx=0x7f6a4800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.511+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a4801d070 con 0x7f6a540ff060 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.511+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5419b080 con 0x7f6a540ff060 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.511+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6a4800f460 con 0x7f6a540ff060 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.511+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a48021620 con 0x7f6a540ff060 2024-05-06T14:27:54.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.511+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5419b570 con 0x7f6a540ff060 2024-05-06T14:27:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.512+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a54107bb0 con 0x7f6a540ff060 2024-05-06T14:27:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.512+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f6a4800f5d0 con 0x7f6a540ff060 2024-05-06T14:27:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.513+0000 7f6a51ffb700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6a4406c680 0x7f6a4406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.513+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f6a4808c720 con 0x7f6a540ff060 2024-05-06T14:27:54.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.513+0000 7f6a53fff700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6a4406c680 0x7f6a4406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:54.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.513+0000 7f6a53fff700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6a4406c680 0x7f6a4406eb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6a54069e50 tx=0x7f6a4000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:54.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.517+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6a4805ace0 con 0x7f6a540ff060 2024-05-06T14:27:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.710+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f6a5419bbb0 con 0x7f6a540ff060 2024-05-06T14:27:54.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.710+0000 7f6a51ffb700 1 -- 172.21.15.153:0/655669085 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f6a4805a870 con 0x7f6a540ff060 2024-05-06T14:27:54.698 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6a4406c680 msgr2=0x7f6a4406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6a4406c680 0x7f6a4406eb30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6a54069e50 tx=0x7f6a4000b410 comp rx=0 tx=0).stop 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 msgr2=0x7f6a540698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540698f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800bda0 tx=0x7f6a4800bdd0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 shutdown_connections 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a540ff060 0x7f6a540698f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6a4406c680 0x7f6a4406eb30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 --2- 172.21.15.153:0/655669085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a54100260 0x7f6a5419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:54.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 >> 172.21.15.153:0/655669085 conn(0x7f6a540fa610 msgr2=0x7f6a54103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 shutdown_connections 2024-05-06T14:27:54.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:54.712+0000 7f6a5acab700 1 -- 172.21.15.153:0/655669085 wait complete. 2024-05-06T14:27:54.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:54 smithi153 ceph-mon[27177]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-05-06T14:27:54.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:54 smithi153 ceph-mon[27177]: osdmap e39: 6 total, 5 up, 6 in 2024-05-06T14:27:54.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:54 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:54.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:54 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:54.859 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:54 smithi190 ceph-mon[33671]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-05-06T14:27:54.860 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:54 smithi190 ceph-mon[33671]: osdmap e39: 6 total, 5 up, 6 in 2024-05-06T14:27:54.860 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:54 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:54.860 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:54 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:55.444 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1715005674,"num_in_osds":6,"osd_in_since":1715005655,"num_remapped_pgs":0} 2024-05-06T14:27:55.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T14:27:55.653 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:27:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: purged_snaps scrub starts 2024-05-06T14:27:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: purged_snaps scrub ok 2024-05-06T14:27:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: osd.5 [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] boot 2024-05-06T14:27:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T14:27:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:55.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/655669085' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T14:27:55.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:55.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:55.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: purged_snaps scrub starts 2024-05-06T14:27:55.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: purged_snaps scrub ok 2024-05-06T14:27:55.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: osd.5 [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] boot 2024-05-06T14:27:55.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T14:27:55.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:27:55.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/655669085' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T14:27:55.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-06T14:27:55.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:27:56.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T14:27:56.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: Detected new or changed devices on smithi190 2024-05-06T14:27:56.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi190 to 4970M 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:56.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:56 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: Detected new or changed devices on smithi190 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:27:56.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi190 to 4970M 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:27:56.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:56 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.363+0000 7fbbd4451700 1 -- 172.21.15.153:0/996580528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 msgr2=0x7fbbcc1007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.363+0000 7fbbd4451700 1 --2- 172.21.15.153:0/996580528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc1007a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009910 tx=0x7fbbbc009c20 comp rx=0 tx=0).stop 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.363+0000 7fbbd4451700 1 -- 172.21.15.153:0/996580528 shutdown_connections 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.363+0000 7fbbd4451700 1 --2- 172.21.15.153:0/996580528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 0x7fbbcc104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.363+0000 7fbbd4451700 1 --2- 172.21.15.153:0/996580528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc1007a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.363+0000 7fbbd4451700 1 -- 172.21.15.153:0/996580528 >> 172.21.15.153:0/996580528 conn(0x7fbbcc0fbc90 msgr2=0x7fbbcc0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.364+0000 7fbbd4451700 1 -- 172.21.15.153:0/996580528 shutdown_connections 2024-05-06T14:27:57.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.364+0000 7fbbd4451700 1 -- 172.21.15.153:0/996580528 wait complete. 2024-05-06T14:27:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.364+0000 7fbbd4451700 1 Processor -- start 2024-05-06T14:27:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.364+0000 7fbbd4451700 1 -- start start 2024-05-06T14:27:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd4451700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:57.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd4451700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 0x7fbbcc1962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd4451700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc196980 con 0x7fbbcc1003d0 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd4451700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc19a710 con 0x7fbbcc100d70 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43428/0 (socket says 172.21.15.153:43428) 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 -- 172.21.15.153:0/2999354150 learned_addr learned my addr 172.21.15.153:0/2999354150 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd19ec700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 0x7fbbcc1962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 -- 172.21.15.153:0/2999354150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 msgr2=0x7fbbcc1962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 0x7fbbcc1962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 -- 172.21.15.153:0/2999354150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc0095f0 con 0x7fbbcc1003d0 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd19ec700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 0x7fbbcc1962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd21ed700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc195d60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc006010 tx=0x7fbbbc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbbc01d070 con 0x7fbbcc1003d0 2024-05-06T14:27:57.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbcc19a990 con 0x7fbbcc1003d0 2024-05-06T14:27:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbbbc004f50 con 0x7fbbcc1003d0 2024-05-06T14:27:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbbc021640 con 0x7fbbcc1003d0 2024-05-06T14:27:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.365+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbcc19ae80 con 0x7fbbcc1003d0 2024-05-06T14:27:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.366+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc1086e0 con 0x7fbbcc1003d0 2024-05-06T14:27:57.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.367+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fbbbc00f460 con 0x7fbbcc1003d0 2024-05-06T14:27:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.367+0000 7fbbc37fe700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbbb806c680 0x7fbbb806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:27:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.367+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbbbc08c770 con 0x7fbbcc1003d0 2024-05-06T14:27:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.367+0000 7fbbd19ec700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbbb806c680 0x7fbbb806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:27:57.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.368+0000 7fbbd19ec700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbbb806c680 0x7fbbb806eb30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8005950 tx=0x7fbbc80058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:27:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.372+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbbbc057190 con 0x7fbbcc1003d0 2024-05-06T14:27:57.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.564+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbcc19b1b0 con 0x7fbbcc1003d0 2024-05-06T14:27:57.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.565+0000 7fbbc37fe700 1 -- 172.21.15.153:0/2999354150 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11364 (secure 0 0 0) 0x7fbbbc00fe60 con 0x7fbbcc1003d0 2024-05-06T14:27:57.552 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:27:57.553 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","created":"2024-05-06T14:22:12.905455+0000","modified":"2024-05-06T14:27:55.592209+0000","last_up_change":"2024-05-06T14:27:54.584488+0000","last_in_change":"2024-05-06T14:27:35.267866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T14:26:41.240977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de83eb29-8176-4eb5-9be6-64b9e5449892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6803","nonce":908509767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6805","nonce":908509767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6809","nonce":908509767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6807","nonce":908509767}]},"public_addr":"172.21.15.153:6803/908509767","cluster_addr":"172.21.15.153:6805/908509767","heartbeat_back_addr":"172.21.15.153:6809/908509767","heartbeat_front_addr":"172.21.15.153:6807/908509767","state":["exists","up"]},{"osd":1,"uuid":"24a1c7ac-a168-4e7d-8c34-3e4c73408756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6811","nonce":3716718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6813","nonce":3716718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6817","nonce":3716718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6815","nonce":3716718209}]},"public_addr":"172.21.15.153:6811/3716718209","cluster_addr":"172.21.15.153:6813/3716718209","heartbeat_back_addr":"172.21.15.153:6817/3716718209","heartbeat_front_addr":"172.21.15.153:6815/3716718209","state":["exists","up"]},{"osd":2,"uuid":"5b89edb2-1c0e-4481-91ed-157d767c0c05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6819","nonce":3169611803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6821","nonce":3169611803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6825","nonce":3169611803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6823","nonce":3169611803}]},"public_addr":"172.21.15.153:6819/3169611803","cluster_addr":"172.21.15.153:6821/3169611803","heartbeat_back_addr":"172.21.15.153:6825/3169611803","heartbeat_front_addr":"172.21.15.153:6823/3169611803","state":["exists","up"]},{"osd":3,"uuid":"74b7823e-2b78-4570-9b7a-89abc9d7fbb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6801","nonce":492034660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6803","nonce":492034660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6807","nonce":492034660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6805","nonce":492034660}]},"public_addr":"172.21.15.190:6801/492034660","cluster_addr":"172.21.15.190:6803/492034660","heartbeat_back_addr":"172.21.15.190:6807/492034660","heartbeat_front_addr":"172.21.15.190:6805/492034660","state":["exists","up"]},{"osd":4,"uuid":"01290c44-3bf1-4da3-bac3-4dff2453cedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6809","nonce":1342651346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6811","nonce":1342651346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6815","nonce":1342651346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6813","nonce":1342651346}]},"public_addr":"172.21.15.190:6809/1342651346","cluster_addr":"172.21.15.190:6811/1342651346","heartbeat_back_addr":"172.21.15.190:6815/1342651346","heartbeat_front_addr":"172.21.15.190:6813/1342651346","state":["exists","up"]},{"osd":5,"uuid":"be366c2e-63d4-4496-b401-772dbe1a2444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6817","nonce":452463735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6819","nonce":452463735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6823","nonce":452463735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6821","nonce":452463735}]},"public_addr":"172.21.15.190:6817/452463735","cluster_addr":"172.21.15.190:6819/452463735","heartbeat_back_addr":"172.21.15.190:6823/452463735","heartbeat_front_addr":"172.21.15.190:6821/452463735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:25:42.510404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:11.276096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:37.855142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:01.189580+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:26.756150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:52.720929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6800/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2433042994":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1460873727":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6801/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1311303452":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6801/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2948592992":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1199411437":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3536681135":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6801/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3158057303":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2161963253":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1309940234":"2024-05-07T14:23:12.780977+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T14:27:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbbb806c680 msgr2=0x7fbbb806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:57.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbbb806c680 0x7fbbb806eb30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8005950 tx=0x7fbbc80058e0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 msgr2=0x7fbbcc195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc195d60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc006010 tx=0x7fbbbc00be20 comp rx=0 tx=0).stop 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 shutdown_connections 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbcc1003d0 0x7fbbcc195d60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbbb806c680 0x7fbbb806eb30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 --2- 172.21.15.153:0/2999354150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbcc100d70 0x7fbbcc1962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 >> 172.21.15.153:0/2999354150 conn(0x7fbbcc0fbc90 msgr2=0x7fbbcc0fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 shutdown_connections 2024-05-06T14:27:57.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:27:57.567+0000 7fbbd4451700 1 -- 172.21.15.153:0/2999354150 wait complete. 2024-05-06T14:27:57.939 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:57 smithi190 ceph-mon[33671]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:27:57.939 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:27:57 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2999354150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:27:58.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:57 smithi153 ceph-mon[27177]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:27:58.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:27:57 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2999354150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:27:58.250 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T14:26:41.240977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-06T14:27:58.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-06T14:27:58.453 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:00.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 -- 172.21.15.153:0/2047980053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 msgr2=0x7fc5340ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:00.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 --2- 172.21.15.153:0/2047980053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc5340ff6a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc524009910 tx=0x7fc524009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:00.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 -- 172.21.15.153:0/2047980053 shutdown_connections 2024-05-06T14:28:00.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 --2- 172.21.15.153:0/2047980053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5340ffc70 0x7fc534103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 --2- 172.21.15.153:0/2047980053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc5340ff6a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 -- 172.21.15.153:0/2047980053 >> 172.21.15.153:0/2047980053 conn(0x7fc5340fabb0 msgr2=0x7fc5340fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:00.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 -- 172.21.15.153:0/2047980053 shutdown_connections 2024-05-06T14:28:00.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 -- 172.21.15.153:0/2047980053 wait complete. 2024-05-06T14:28:00.207 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:00 smithi153 ceph-mon[27177]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:00.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.217+0000 7fc5397cd700 1 Processor -- start 2024-05-06T14:28:00.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc5397cd700 1 -- start start 2024-05-06T14:28:00.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc5397cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc534193990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc5397cd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5340ffc70 0x7fc534193ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc5397cd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc534195ab0 con 0x7fc5340ff2d0 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc5397cd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc534195bf0 con 0x7fc5340ffc70 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc534193990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc534193990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48350/0 (socket says 172.21.15.153:48350) 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 -- 172.21.15.153:0/1040257584 learned_addr learned my addr 172.21.15.153:0/1040257584 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:00.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 -- 172.21.15.153:0/1040257584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5340ffc70 msgr2=0x7fc534193ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc52bfff700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5340ffc70 0x7fc534193ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5340ffc70 0x7fc534193ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 -- 172.21.15.153:0/1040257584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240095f0 con 0x7fc5340ff2d0 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc532ffd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc534193990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc524006010 tx=0x7fc52400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.218+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52401d070 con 0x7fc5340ff2d0 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.219+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc524004f50 con 0x7fc5340ff2d0 2024-05-06T14:28:00.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.219+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc534194550 con 0x7fc5340ff2d0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.219+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc524021640 con 0x7fc5340ff2d0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.219+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5341a6780 con 0x7fc5340ff2d0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.220+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53404ea50 con 0x7fc5340ff2d0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.220+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fc52400f460 con 0x7fc5340ff2d0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.220+0000 7fc530ff9700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc52006c680 0x7fc52006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.221+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc52408c770 con 0x7fc5340ff2d0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.221+0000 7fc52bfff700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc52006c680 0x7fc52006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:00.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.221+0000 7fc52bfff700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc52006c680 0x7fc52006eb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc51c005950 tx=0x7fc51c0058e0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:00.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.225+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc524057190 con 0x7fc5340ff2d0 2024-05-06T14:28:00.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.412+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5341a6a40 con 0x7fc5340ff2d0 2024-05-06T14:28:00.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.412+0000 7fc530ff9700 1 -- 172.21.15.153:0/1040257584 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc52400fe60 con 0x7fc5340ff2d0 2024-05-06T14:28:00.400 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-05-06T14:28:00.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc52006c680 msgr2=0x7fc52006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc52006c680 0x7fc52006eb30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc51c005950 tx=0x7fc51c0058e0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 msgr2=0x7fc534193990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc534193990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc524006010 tx=0x7fc52400be20 comp rx=0 tx=0).stop 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 shutdown_connections 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5340ff2d0 0x7fc534193990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc52006c680 0x7fc52006eb30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 --2- 172.21.15.153:0/1040257584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5340ffc70 0x7fc534193ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 >> 172.21.15.153:0/1040257584 conn(0x7fc5340fabb0 msgr2=0x7fc534068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 shutdown_connections 2024-05-06T14:28:00.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:00.415+0000 7fc5397cd700 1 -- 172.21.15.153:0/1040257584 wait complete. 2024-05-06T14:28:00.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:00 smithi190 ceph-mon[33671]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:01.078 INFO:tasks.cephadm:Setting up client nodes... 2024-05-06T14:28:01.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-06T14:28:01.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:01 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1040257584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:28:01.281 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:01 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1040257584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:28:01.282 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:02.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:02 smithi153 ceph-mon[27177]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:02.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:02 smithi190 ceph-mon[33671]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 -- 172.21.15.153:0/1174617148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 msgr2=0x7f20dc0fe8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 --2- 172.21.15.153:0/1174617148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc0fe8b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009910 tx=0x7f20cc009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 -- 172.21.15.153:0/1174617148 shutdown_connections 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 --2- 172.21.15.153:0/1174617148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20dc101520 0x7f20dc101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 --2- 172.21.15.153:0/1174617148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc0fe8b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 -- 172.21.15.153:0/1174617148 >> 172.21.15.153:0/1174617148 conn(0x7f20dc0f9dd0 msgr2=0x7f20dc0fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 -- 172.21.15.153:0/1174617148 shutdown_connections 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.993+0000 7f20e2921700 1 -- 172.21.15.153:0/1174617148 wait complete. 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20e2921700 1 Processor -- start 2024-05-06T14:28:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20e2921700 1 -- start start 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20e2921700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20e2921700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20dc101520 0x7f20dc195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20e2921700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc196580 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20e2921700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc1966c0 con 0x7f20dc101520 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48372/0 (socket says 172.21.15.153:48372) 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20dbfff700 1 -- 172.21.15.153:0/507388532 learned_addr learned my addr 172.21.15.153:0/507388532 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20dbfff700 1 -- 172.21.15.153:0/507388532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20dc101520 msgr2=0x7f20dc195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20dbfff700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20dc101520 0x7f20dc195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.994+0000 7f20dbfff700 1 -- 172.21.15.153:0/507388532 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc0095f0 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.995+0000 7f20dbfff700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc195a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0092e0 tx=0x7f20cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.995+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20cc01d070 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.995+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc19b110 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.995+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f20cc022470 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.995+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20cc00f690 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.995+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc19b600 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.996+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc108d00 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.996+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f20cc0225e0 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.997+0000 7f20d97fa700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f20c406c680 0x7f20c406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.997+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f20cc08cb80 con 0x7f20dc0fe4a0 2024-05-06T14:28:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.997+0000 7f20db7fe700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f20c406c680 0x7f20c406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:02.997+0000 7f20db7fe700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f20c406c680 0x7f20c406eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20d000a990 tx=0x7f20d0005c80 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:02.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.001+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f20cc05b0e0 con 0x7f20dc0fe4a0 2024-05-06T14:28:03.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.271+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f20dc108ec0 con 0x7f20dc0fe4a0 2024-05-06T14:28:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.275+0000 7f20d97fa700 1 -- 172.21.15.153:0/507388532 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f20cc027050 con 0x7f20dc0fe4a0 2024-05-06T14:28:03.262 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-05-06T14:28:03.263 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDz6DhmRu5BEBAAnWBs0kqyN8+QahitucMJMw== 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.277+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f20c406c680 msgr2=0x7f20c406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.277+0000 7f20e2921700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f20c406c680 0x7f20c406eb30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20d000a990 tx=0x7f20d0005c80 comp rx=0 tx=0).stop 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.277+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 msgr2=0x7f20dc195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.277+0000 7f20e2921700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc195a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0092e0 tx=0x7f20cc00be20 comp rx=0 tx=0).stop 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 shutdown_connections 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20dc0fe4a0 0x7f20dc195a20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f20c406c680 0x7f20c406eb30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:03.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 --2- 172.21.15.153:0/507388532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20dc101520 0x7f20dc195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:03.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 >> 172.21.15.153:0/507388532 conn(0x7f20dc0f9dd0 msgr2=0x7f20dc104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:03.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 shutdown_connections 2024-05-06T14:28:03.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:03.278+0000 7f20e2921700 1 -- 172.21.15.153:0/507388532 wait complete. 2024-05-06T14:28:03.820 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:28:03.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-06T14:28:03.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-06T14:28:03.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-06T14:28:04.079 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi190/config 2024-05-06T14:28:04.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:04 smithi190 ceph-mon[33671]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:04 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/507388532' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T14:28:04.186 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:04 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/507388532' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T14:28:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:04 smithi153 ceph-mon[27177]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:04 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/507388532' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T14:28:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:04 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/507388532' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T14:28:05.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 -- 172.21.15.190:0/1107149947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f0101440 msgr2=0x7fc2f0101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:05.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 --2- 172.21.15.190:0/1107149947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f0101440 0x7fc2f0101890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009910 tx=0x7fc2ec009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:05.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 -- 172.21.15.190:0/1107149947 shutdown_connections 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 --2- 172.21.15.190:0/1107149947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f0101440 0x7fc2f0101890 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 --2- 172.21.15.190:0/1107149947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00fe460 0x7fc2f00fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 -- 172.21.15.190:0/1107149947 >> 172.21.15.190:0/1107149947 conn(0x7fc2f00f9db0 msgr2=0x7fc2f00fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 -- 172.21.15.190:0/1107149947 shutdown_connections 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.731+0000 7fc2f728f700 1 -- 172.21.15.190:0/1107149947 wait complete. 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f728f700 1 Processor -- start 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f728f700 1 -- start start 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f728f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 0x7fc2f0195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f728f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f0101440 0x7fc2f0195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f728f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f01964f0 con 0x7fc2f0101440 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f728f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0196630 con 0x7fc2f00fe460 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f502b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 0x7fc2f0195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f502b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 0x7fc2f0195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:60980/0 (socket says 172.21.15.190:60980) 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f502b700 1 -- 172.21.15.190:0/2478911240 learned_addr learned my addr 172.21.15.190:0/2478911240 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f482a700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f0101440 0x7fc2f0195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f502b700 1 -- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f0101440 msgr2=0x7fc2f0195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f502b700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f0101440 0x7fc2f0195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.732+0000 7fc2f502b700 1 -- 172.21.15.190:0/2478911240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0095f0 con 0x7fc2f00fe460 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.733+0000 7fc2f502b700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 0x7fc2f0195990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400d900 tx=0x7fc2e400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.733+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e40041f0 con 0x7fc2f00fe460 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.733+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f019b0e0 con 0x7fc2f00fe460 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.733+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc2e4004350 con 0x7fc2f00fe460 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.733+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e4003da0 con 0x7fc2f00fe460 2024-05-06T14:28:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.733+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f019b630 con 0x7fc2f00fe460 2024-05-06T14:28:05.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.734+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0108ce0 con 0x7fc2f00fe460 2024-05-06T14:28:05.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.734+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fc2e4009730 con 0x7fc2f00fe460 2024-05-06T14:28:05.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.735+0000 7fc2e27fc700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc2dc06c680 0x7fc2dc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:05.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.735+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc2e408b520 con 0x7fc2f00fe460 2024-05-06T14:28:05.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.735+0000 7fc2f482a700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc2dc06c680 0x7fc2dc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:05.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.735+0000 7fc2f482a700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc2dc06c680 0x7fc2dc06eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec005b40 tx=0x7fc2ec005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:05.728 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:05.741+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc2e40599d0 con 0x7fc2f00fe460 2024-05-06T14:28:05.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.002+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc2f019bd30 con 0x7fc2f00fe460 2024-05-06T14:28:05.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.007+0000 7fc2e27fc700 1 -- 172.21.15.190:0/2478911240 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc2e4059560 con 0x7fc2f00fe460 2024-05-06T14:28:05.995 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-05-06T14:28:05.995 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD26DhmWBxKABAAbTgmpx1sXs0IsQAxv52fKQ== 2024-05-06T14:28:05.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc2dc06c680 msgr2=0x7fc2dc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:05.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc2dc06c680 0x7fc2dc06eb30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec005b40 tx=0x7fc2ec005a90 comp rx=0 tx=0).stop 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 msgr2=0x7fc2f0195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 0x7fc2f0195990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400d900 tx=0x7fc2e400dcc0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 shutdown_connections 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc2dc06c680 0x7fc2dc06eb30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f00fe460 0x7fc2f0195990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 --2- 172.21.15.190:0/2478911240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f0101440 0x7fc2f0195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 >> 172.21.15.190:0/2478911240 conn(0x7fc2f00f9db0 msgr2=0x7fc2f0104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.009+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 shutdown_connections 2024-05-06T14:28:05.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:28:06.010+0000 7fc2f728f700 1 -- 172.21.15.190:0/2478911240 wait complete. 2024-05-06T14:28:06.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:06 smithi153 ceph-mon[27177]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:06.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:06 smithi153 ceph-mon[27177]: from='client.? 172.21.15.190:0/2478911240' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T14:28:06.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:06 smithi153 ceph-mon[27177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T14:28:06.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:06 smithi153 ceph-mon[27177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T14:28:06.304 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:06 smithi190 ceph-mon[33671]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:06.304 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:06 smithi190 ceph-mon[33671]: from='client.? 172.21.15.190:0/2478911240' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T14:28:06.304 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:06 smithi190 ceph-mon[33671]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T14:28:06.304 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:06 smithi190 ceph-mon[33671]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T14:28:06.618 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:28:06.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-06T14:28:06.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-06T14:28:06.653 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-06T14:28:06.654 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-06T14:28:06.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-06T14:28:06.875 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:08.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:08 smithi153 ceph-mon[27177]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:08.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:08 smithi190 ceph-mon[33671]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 -- 172.21.15.153:0/708525875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ffd30 msgr2=0x7fecb4103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 --2- 172.21.15.153:0/708525875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ffd30 0x7fecb4103ce0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feca8009910 tx=0x7feca8009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 -- 172.21.15.153:0/708525875 shutdown_connections 2024-05-06T14:28:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 --2- 172.21.15.153:0/708525875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ffd30 0x7fecb4103ce0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:08.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 --2- 172.21.15.153:0/708525875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb40ff390 0x7fecb40ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 -- 172.21.15.153:0/708525875 >> 172.21.15.153:0/708525875 conn(0x7fecb4074f50 msgr2=0x7fecb4075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 -- 172.21.15.153:0/708525875 shutdown_connections 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.589+0000 7fecba990700 1 -- 172.21.15.153:0/708525875 wait complete. 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecba990700 1 Processor -- start 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecba990700 1 -- start start 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecba990700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 0x7fecb4195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecba990700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb40ffd30 0x7fecb41962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecba990700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb41969b0 con 0x7fecb40ff390 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecba990700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb419a740 con 0x7fecb40ffd30 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 0x7fecb4195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 0x7fecb4195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48388/0 (socket says 172.21.15.153:48388) 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb3fff700 1 -- 172.21.15.153:0/563596214 learned_addr learned my addr 172.21.15.153:0/563596214 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb37fe700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb40ffd30 0x7fecb41962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb3fff700 1 -- 172.21.15.153:0/563596214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb40ffd30 msgr2=0x7fecb41962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:08.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb3fff700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb40ffd30 0x7fecb41962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.590+0000 7fecb3fff700 1 -- 172.21.15.153:0/563596214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80095f0 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.591+0000 7fecb3fff700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 0x7fecb4195d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7feca400d8d0 tx=0x7feca400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.591+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feca4009940 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.591+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feca4010460 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.591+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb419aa20 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.591+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feca400f5f0 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.591+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb419af70 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.592+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb404ea50 con 0x7fecb40ff390 2024-05-06T14:28:08.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.592+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7feca40105d0 con 0x7fecb40ff390 2024-05-06T14:28:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.593+0000 7fecb17fa700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fec9c06c680 0x7fec9c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.593+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7feca408b540 con 0x7fecb40ff390 2024-05-06T14:28:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.593+0000 7fecb37fe700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fec9c06c680 0x7fec9c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:08.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.593+0000 7fecb37fe700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fec9c06c680 0x7fec9c06eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fecb41973b0 tx=0x7feca801a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:08.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.597+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7feca4055f60 con 0x7fecb40ff390 2024-05-06T14:28:08.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.843+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fecb41970f0 con 0x7fecb40ff390 2024-05-06T14:28:08.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.846+0000 7fecb17fa700 1 -- 172.21.15.153:0/563596214 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+172864 (secure 0 0 0) 0x7feca4059580 con 0x7fecb40ff390 2024-05-06T14:28:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:28:08.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.848+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fec9c06c680 msgr2=0x7fec9c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.848+0000 7fecba990700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fec9c06c680 0x7fec9c06eb30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fecb41973b0 tx=0x7feca801a040 comp rx=0 tx=0).stop 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.848+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 msgr2=0x7fecb4195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.848+0000 7fecba990700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 0x7fecb4195d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7feca400d8d0 tx=0x7feca400dc90 comp rx=0 tx=0).stop 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 shutdown_connections 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecb40ff390 0x7fecb4195d90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fec9c06c680 0x7fec9c06eb30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 --2- 172.21.15.153:0/563596214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecb40ffd30 0x7fecb41962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 >> 172.21.15.153:0/563596214 conn(0x7fecb4074f50 msgr2=0x7fecb4103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 shutdown_connections 2024-05-06T14:28:08.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:08.849+0000 7fecba990700 1 -- 172.21.15.153:0/563596214 wait complete. 2024-05-06T14:28:09.236 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:09 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/563596214' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T14:28:09.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:09 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/563596214' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T14:28:09.596 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":19,"active_gid":14217,"active_name":"smithi153.dolfed","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1473246849},{"type":"v1","addr":"172.21.15.153:6801","nonce":1473246849}]},"active_addr":"172.21.15.153:6801/1473246849","active_change":"2024-05-06T14:24:24.923763+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14242,"name":"smithi190.qauohf","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.153:8443/","prometheus":"http://172.21.15.153:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3898448400}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1418596404}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":4101428691}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3340397168}]}]} 2024-05-06T14:28:09.600 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-06T14:28:09.600 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-06T14:28:09.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T14:28:09.809 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:10.240 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:10 smithi153 ceph-mon[27177]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:10.240 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:10.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:10 smithi190 ceph-mon[33671]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:10.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:11.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1942974320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068fd0 msgr2=0x7fa388104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1942974320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068fd0 0x7fa388104df0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009910 tx=0x7fa37c009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1942974320 shutdown_connections 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1942974320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068fd0 0x7fa388104df0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1942974320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa388068630 0x7fa388068a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1942974320 >> 172.21.15.153:0/1942974320 conn(0x7fa388074f50 msgr2=0x7fa388075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1942974320 shutdown_connections 2024-05-06T14:28:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.436+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1942974320 wait complete. 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38cbe6700 1 Processor -- start 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38cbe6700 1 -- start start 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38cbe6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 0x7fa3880ff200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38cbe6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa388068fd0 0x7fa3880ff740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38cbe6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880ffc80 con 0x7fa388068630 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38cbe6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880ffdc0 con 0x7fa388068fd0 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 0x7fa3880ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 0x7fa3880ff200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40532/0 (socket says 172.21.15.153:40532) 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa38659c700 1 -- 172.21.15.153:0/1650592543 learned_addr learned my addr 172.21.15.153:0/1650592543 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.437+0000 7fa385d9b700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa388068fd0 0x7fa3880ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa38659c700 1 -- 172.21.15.153:0/1650592543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa388068fd0 msgr2=0x7fa3880ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa38659c700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa388068fd0 0x7fa3880ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa38659c700 1 -- 172.21.15.153:0/1650592543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0095f0 con 0x7fa388068630 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa38659c700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 0x7fa3880ff200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa38000ba50 tx=0x7fa38000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa38000c900 con 0x7fa388068630 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa380014440 con 0x7fa388068630 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3800075a0 con 0x7fa388068630 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3881a1ff0 con 0x7fa388068630 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.438+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3881a2400 con 0x7fa388068630 2024-05-06T14:28:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.439+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3881086d0 con 0x7fa388068630 2024-05-06T14:28:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.439+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa3800145b0 con 0x7fa388068630 2024-05-06T14:28:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.440+0000 7fa3777fe700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa37006c680 0x7fa37006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.440+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa380006e40 con 0x7fa388068630 2024-05-06T14:28:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.440+0000 7fa385d9b700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa37006c680 0x7fa37006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:11.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.440+0000 7fa385d9b700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa37006c680 0x7fa37006eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3881009f0 tx=0x7fa37c01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.444+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa38004e720 con 0x7fa388068630 2024-05-06T14:28:11.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.635+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa38804ea50 con 0x7fa388068630 2024-05-06T14:28:11.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.636+0000 7fa3777fe700 1 -- 172.21.15.153:0/1650592543 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11364 (secure 0 0 0) 0x7fa3800556e0 con 0x7fa388068630 2024-05-06T14:28:11.623 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:28:11.623 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","created":"2024-05-06T14:22:12.905455+0000","modified":"2024-05-06T14:27:55.592209+0000","last_up_change":"2024-05-06T14:27:54.584488+0000","last_in_change":"2024-05-06T14:27:35.267866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T14:26:41.240977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de83eb29-8176-4eb5-9be6-64b9e5449892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6803","nonce":908509767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6805","nonce":908509767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6809","nonce":908509767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6807","nonce":908509767}]},"public_addr":"172.21.15.153:6803/908509767","cluster_addr":"172.21.15.153:6805/908509767","heartbeat_back_addr":"172.21.15.153:6809/908509767","heartbeat_front_addr":"172.21.15.153:6807/908509767","state":["exists","up"]},{"osd":1,"uuid":"24a1c7ac-a168-4e7d-8c34-3e4c73408756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6811","nonce":3716718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6813","nonce":3716718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6817","nonce":3716718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6815","nonce":3716718209}]},"public_addr":"172.21.15.153:6811/3716718209","cluster_addr":"172.21.15.153:6813/3716718209","heartbeat_back_addr":"172.21.15.153:6817/3716718209","heartbeat_front_addr":"172.21.15.153:6815/3716718209","state":["exists","up"]},{"osd":2,"uuid":"5b89edb2-1c0e-4481-91ed-157d767c0c05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6819","nonce":3169611803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6821","nonce":3169611803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6825","nonce":3169611803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6823","nonce":3169611803}]},"public_addr":"172.21.15.153:6819/3169611803","cluster_addr":"172.21.15.153:6821/3169611803","heartbeat_back_addr":"172.21.15.153:6825/3169611803","heartbeat_front_addr":"172.21.15.153:6823/3169611803","state":["exists","up"]},{"osd":3,"uuid":"74b7823e-2b78-4570-9b7a-89abc9d7fbb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6801","nonce":492034660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6803","nonce":492034660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6807","nonce":492034660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6805","nonce":492034660}]},"public_addr":"172.21.15.190:6801/492034660","cluster_addr":"172.21.15.190:6803/492034660","heartbeat_back_addr":"172.21.15.190:6807/492034660","heartbeat_front_addr":"172.21.15.190:6805/492034660","state":["exists","up"]},{"osd":4,"uuid":"01290c44-3bf1-4da3-bac3-4dff2453cedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6809","nonce":1342651346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6811","nonce":1342651346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6815","nonce":1342651346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6813","nonce":1342651346}]},"public_addr":"172.21.15.190:6809/1342651346","cluster_addr":"172.21.15.190:6811/1342651346","heartbeat_back_addr":"172.21.15.190:6815/1342651346","heartbeat_front_addr":"172.21.15.190:6813/1342651346","state":["exists","up"]},{"osd":5,"uuid":"be366c2e-63d4-4496-b401-772dbe1a2444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6817","nonce":452463735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6819","nonce":452463735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6823","nonce":452463735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6821","nonce":452463735}]},"public_addr":"172.21.15.190:6817/452463735","cluster_addr":"172.21.15.190:6819/452463735","heartbeat_back_addr":"172.21.15.190:6823/452463735","heartbeat_front_addr":"172.21.15.190:6821/452463735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:25:42.510404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:11.276096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:37.855142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:01.189580+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:26.756150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:52.720929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6800/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2433042994":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1460873727":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6801/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1311303452":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6801/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2948592992":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1199411437":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3536681135":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6801/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3158057303":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2161963253":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1309940234":"2024-05-07T14:23:12.780977+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T14:28:11.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa37006c680 msgr2=0x7fa37006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:11.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa37006c680 0x7fa37006eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3881009f0 tx=0x7fa37c01a040 comp rx=0 tx=0).stop 2024-05-06T14:28:11.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 msgr2=0x7fa3880ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:11.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 0x7fa3880ff200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa38000ba50 tx=0x7fa38000bd60 comp rx=0 tx=0).stop 2024-05-06T14:28:11.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 shutdown_connections 2024-05-06T14:28:11.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa388068630 0x7fa3880ff200 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:11.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa37006c680 0x7fa37006eb30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:11.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 --2- 172.21.15.153:0/1650592543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa388068fd0 0x7fa3880ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:11.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 >> 172.21.15.153:0/1650592543 conn(0x7fa388074f50 msgr2=0x7fa3880fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:11.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 shutdown_connections 2024-05-06T14:28:11.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:11.638+0000 7fa38cbe6700 1 -- 172.21.15.153:0/1650592543 wait complete. 2024-05-06T14:28:12.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:12 smithi153 ceph-mon[27177]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:12.242 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:12 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1650592543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:28:12.265 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-06T14:28:12.265 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T14:28:12.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:12 smithi190 ceph-mon[33671]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:12.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:12 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1650592543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:28:12.469 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:14.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 -- 172.21.15.153:0/3219752764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19781013f0 msgr2=0x7f19781017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:14.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 --2- 172.21.15.153:0/3219752764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19781013f0 0x7f19781017c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f196c009910 tx=0x7f196c009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:14.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 -- 172.21.15.153:0/3219752764 shutdown_connections 2024-05-06T14:28:14.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 --2- 172.21.15.153:0/3219752764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1978101d00 0x7f197810a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:14.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 --2- 172.21.15.153:0/3219752764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19781013f0 0x7f19781017c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:14.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 -- 172.21.15.153:0/3219752764 >> 172.21.15.153:0/3219752764 conn(0x7f19780fabd0 msgr2=0x7f19780fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.098+0000 7f197e496700 1 -- 172.21.15.153:0/3219752764 shutdown_connections 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 -- 172.21.15.153:0/3219752764 wait complete. 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 Processor -- start 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 -- start start 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19781013f0 0x7f197819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 0x7f197819c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197819c600 con 0x7f1978101d00 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.099+0000 7f197e496700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1978195b40 con 0x7f19781013f0 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 0x7f197819c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 0x7f197819c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40556/0 (socket says 172.21.15.153:40556) 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 -- 172.21.15.153:0/636705169 learned_addr learned my addr 172.21.15.153:0/636705169 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:14.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f1977fff700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19781013f0 0x7f197819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:14.089 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:14 smithi153 ceph-mon[27177]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:14.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 -- 172.21.15.153:0/636705169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19781013f0 msgr2=0x7f197819bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:14.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19781013f0 0x7f197819bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:14.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 -- 172.21.15.153:0/636705169 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c0095f0 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19777fe700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 0x7f197819c000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f196800c930 tx=0x7f196800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1968007ad0 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1978195e20 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f196800ce80 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1968018630 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.100+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1978196370 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.101+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19781078d0 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.102+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f1968007c30 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.103+0000 7f19757fa700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f196006c680 0x7f196006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.103+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f196808b5e0 con 0x7f1978101d00 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.103+0000 7f1977fff700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f196006c680 0x7f196006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.103+0000 7f1977fff700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f196006c680 0x7f196006eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f196c006010 tx=0x7f196c01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:14.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.107+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f19680599f0 con 0x7f1978101d00 2024-05-06T14:28:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.300+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f197804ea50 con 0x7f1978101d00 2024-05-06T14:28:14.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.301+0000 7f19757fa700 1 -- 172.21.15.153:0/636705169 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11364 (secure 0 0 0) 0x7f1968059580 con 0x7f1978101d00 2024-05-06T14:28:14.289 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:28:14.289 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","created":"2024-05-06T14:22:12.905455+0000","modified":"2024-05-06T14:27:55.592209+0000","last_up_change":"2024-05-06T14:27:54.584488+0000","last_in_change":"2024-05-06T14:27:35.267866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T14:26:41.240977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de83eb29-8176-4eb5-9be6-64b9e5449892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6803","nonce":908509767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6805","nonce":908509767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6809","nonce":908509767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6807","nonce":908509767}]},"public_addr":"172.21.15.153:6803/908509767","cluster_addr":"172.21.15.153:6805/908509767","heartbeat_back_addr":"172.21.15.153:6809/908509767","heartbeat_front_addr":"172.21.15.153:6807/908509767","state":["exists","up"]},{"osd":1,"uuid":"24a1c7ac-a168-4e7d-8c34-3e4c73408756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6811","nonce":3716718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6813","nonce":3716718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6817","nonce":3716718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6815","nonce":3716718209}]},"public_addr":"172.21.15.153:6811/3716718209","cluster_addr":"172.21.15.153:6813/3716718209","heartbeat_back_addr":"172.21.15.153:6817/3716718209","heartbeat_front_addr":"172.21.15.153:6815/3716718209","state":["exists","up"]},{"osd":2,"uuid":"5b89edb2-1c0e-4481-91ed-157d767c0c05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6819","nonce":3169611803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6821","nonce":3169611803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6825","nonce":3169611803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6823","nonce":3169611803}]},"public_addr":"172.21.15.153:6819/3169611803","cluster_addr":"172.21.15.153:6821/3169611803","heartbeat_back_addr":"172.21.15.153:6825/3169611803","heartbeat_front_addr":"172.21.15.153:6823/3169611803","state":["exists","up"]},{"osd":3,"uuid":"74b7823e-2b78-4570-9b7a-89abc9d7fbb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6801","nonce":492034660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6803","nonce":492034660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6807","nonce":492034660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6805","nonce":492034660}]},"public_addr":"172.21.15.190:6801/492034660","cluster_addr":"172.21.15.190:6803/492034660","heartbeat_back_addr":"172.21.15.190:6807/492034660","heartbeat_front_addr":"172.21.15.190:6805/492034660","state":["exists","up"]},{"osd":4,"uuid":"01290c44-3bf1-4da3-bac3-4dff2453cedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6809","nonce":1342651346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6811","nonce":1342651346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6815","nonce":1342651346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6813","nonce":1342651346}]},"public_addr":"172.21.15.190:6809/1342651346","cluster_addr":"172.21.15.190:6811/1342651346","heartbeat_back_addr":"172.21.15.190:6815/1342651346","heartbeat_front_addr":"172.21.15.190:6813/1342651346","state":["exists","up"]},{"osd":5,"uuid":"be366c2e-63d4-4496-b401-772dbe1a2444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6817","nonce":452463735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6819","nonce":452463735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6823","nonce":452463735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6821","nonce":452463735}]},"public_addr":"172.21.15.190:6817/452463735","cluster_addr":"172.21.15.190:6819/452463735","heartbeat_back_addr":"172.21.15.190:6823/452463735","heartbeat_front_addr":"172.21.15.190:6821/452463735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:25:42.510404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:11.276096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:37.855142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:01.189580+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:26.756150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:52.720929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6800/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2433042994":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1460873727":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6801/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1311303452":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6801/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2948592992":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1199411437":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3536681135":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6801/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3158057303":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2161963253":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1309940234":"2024-05-07T14:23:12.780977+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T14:28:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.303+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f196006c680 msgr2=0x7f196006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:14.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.303+0000 7f197e496700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f196006c680 0x7f196006eb30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f196c006010 tx=0x7f196c01a040 comp rx=0 tx=0).stop 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.303+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 msgr2=0x7f197819c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.303+0000 7f197e496700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 0x7f197819c000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f196800c930 tx=0x7f196800cc40 comp rx=0 tx=0).stop 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 shutdown_connections 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f196006c680 0x7f196006eb30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19781013f0 0x7f197819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 --2- 172.21.15.153:0/636705169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1978101d00 0x7f197819c000 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 >> 172.21.15.153:0/636705169 conn(0x7f19780fabd0 msgr2=0x7f1978104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 shutdown_connections 2024-05-06T14:28:14.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:14.304+0000 7f197e496700 1 -- 172.21.15.153:0/636705169 wait complete. 2024-05-06T14:28:14.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:14 smithi190 ceph-mon[33671]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:15.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-06T14:28:15.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-06T14:28:15.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-06T14:28:15.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-06T14:28:15.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-06T14:28:15.069 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-06T14:28:15.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:15 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/636705169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:28:15.331 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:15.337 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:15.395 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:15.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:15 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/636705169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:28:16.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:16 smithi153 ceph-mon[27177]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:16.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:16 smithi190 ceph-mon[33671]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:18 smithi153 ceph-mon[27177]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:18.438 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:18 smithi190 ceph-mon[33671]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:19.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 -- 172.21.15.153:0/2213714083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 msgr2=0x7f6108104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 --2- 172.21.15.153:0/2213714083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108104d70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009910 tx=0x7f60fc009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 -- 172.21.15.153:0/2213714083 shutdown_connections 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 --2- 172.21.15.153:0/2213714083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108104d70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 --2- 172.21.15.153:0/2213714083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6108100400 0x7f61081007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 -- 172.21.15.153:0/2213714083 >> 172.21.15.153:0/2213714083 conn(0x7f61080fbca0 msgr2=0x7f61080fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 -- 172.21.15.153:0/2213714083 shutdown_connections 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.675+0000 7f610e258700 1 -- 172.21.15.153:0/2213714083 wait complete. 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f610e258700 1 Processor -- start 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f610e258700 1 -- start start 2024-05-06T14:28:19.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f610e258700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6108100400 0x7f6108073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f610e258700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f610e258700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6108073ad0 con 0x7f6108100da0 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f610e258700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6108073c10 con 0x7f6108100400 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f6106ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f6106ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55802/0 (socket says 172.21.15.153:55802) 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f6106ffd700 1 -- 172.21.15.153:0/2064981960 learned_addr learned my addr 172.21.15.153:0/2064981960 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:19.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.676+0000 7f61077fe700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6108100400 0x7f6108073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6106ffd700 1 -- 172.21.15.153:0/2064981960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6108100400 msgr2=0x7f6108073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6106ffd700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6108100400 0x7f6108073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6106ffd700 1 -- 172.21.15.153:0/2064981960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc0095f0 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6106ffd700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108073590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0092e0 tx=0x7f60fc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60fc01d070 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f60fc004fa0 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6108073e90 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60fc00f690 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.677+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61081a2250 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.678+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f610810d5b0 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f60fc022470 con 0x7f6108100da0 2024-05-06T14:28:19.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6104ff9700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f60f006c680 0x7f60f006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f60fc08cca0 con 0x7f6108100da0 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f61077fe700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f60f006c680 0x7f60f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6104ff9700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] conn(0x7f60f0072350 0x7f60f0074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 --> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60f0074e10 con 0x7f60f0072350 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60fc092380 con 0x7f6108100da0 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.679+0000 7f6107fff700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] conn(0x7f60f0072350 0x7f60f0074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.680+0000 7f61077fe700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f60f006c680 0x7f60f006eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f60f8005950 tx=0x7f60f800a300 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.680+0000 7f6107fff700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] conn(0x7f60f0072350 0x7f60f0074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:19.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.680+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== osd.0 v2:172.21.15.153:6802/908509767 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f60f0074e10 con 0x7f60f0072350 2024-05-06T14:28:19.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 --> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f610804ea50 con 0x7f60f0072350 2024-05-06T14:28:19.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f6104ff9700 1 -- 172.21.15.153:0/2064981960 <== osd.0 v2:172.21.15.153:6802/908509767 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f610804ea50 con 0x7f60f0072350 2024-05-06T14:28:19.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] conn(0x7f60f0072350 msgr2=0x7f60f0074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:19.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] conn(0x7f60f0072350 0x7f60f0074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f60f006c680 msgr2=0x7f60f006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f60f006c680 0x7f60f006eb30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f60f8005950 tx=0x7f60f800a300 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 msgr2=0x7f6108073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.696+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108073590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0092e0 tx=0x7f60fc00bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 shutdown_connections 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6802/908509767,v1:172.21.15.153:6803/908509767] conn(0x7f60f0072350 0x7f60f0074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f60f006c680 0x7f60f006eb30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6108100400 0x7f6108073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 --2- 172.21.15.153:0/2064981960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6108100da0 0x7f6108073590 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 >> 172.21.15.153:0/2064981960 conn(0x7f61080fbca0 msgr2=0x7f61080fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 shutdown_connections 2024-05-06T14:28:19.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:19.697+0000 7f610e258700 1 -- 172.21.15.153:0/2064981960 wait complete. 2024-05-06T14:28:20.390 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:20 smithi153 ceph-mon[27177]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:20.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:20 smithi190 ceph-mon[33671]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:22.172 INFO:teuthology.orchestra.run.smithi153.stdout:38654705699 2024-05-06T14:28:22.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-06T14:28:22.336 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:22 smithi153 ceph-mon[27177]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/290156101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ffd30 msgr2=0x7fb6f8103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/290156101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8103ce0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009910 tx=0x7fb6ec009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/290156101 shutdown_connections 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/290156101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8103ce0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/290156101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ff390 0x7fb6f80ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/290156101 >> 172.21.15.153:0/290156101 conn(0x7fb6f8074f50 msgr2=0x7fb6f8075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/290156101 shutdown_connections 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.364+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/290156101 wait complete. 2024-05-06T14:28:22.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6fe4a6700 1 Processor -- start 2024-05-06T14:28:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6fe4a6700 1 -- start start 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6fe4a6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 0x7fb6f8194970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6fe4a6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8194eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6fe4a6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f81953f0 con 0x7fb6f80ff390 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6fe4a6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f8195560 con 0x7fb6f80ffd30 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 0x7fb6f8194970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 0x7fb6f8194970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55834/0 (socket says 172.21.15.153:55834) 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6f7fff700 1 -- 172.21.15.153:0/1298121861 learned_addr learned my addr 172.21.15.153:0/1298121861 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:22.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.365+0000 7fb6f77fe700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8194eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f7fff700 1 -- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ffd30 msgr2=0x7fb6f8194eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f7fff700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8194eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f7fff700 1 -- 172.21.15.153:0/1298121861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec0095f0 con 0x7fb6f80ff390 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f77fe700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8194eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f7fff700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 0x7fb6f8194970 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800d900 tx=0x7fb6e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e80041f0 con 0x7fb6f80ff390 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f81a1fe0 con 0x7fb6f80ff390 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb6e8004ce0 con 0x7fb6f80ff390 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e800b750 con 0x7fb6f80ff390 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.366+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f81a2340 con 0x7fb6f80ff390 2024-05-06T14:28:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.367+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6f810c560 con 0x7fb6f80ff390 2024-05-06T14:28:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb6e800bbd0 con 0x7fb6f80ff390 2024-05-06T14:28:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f57fa700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb6e006c680 0x7fb6e006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb6e808b630 con 0x7fb6f80ff390 2024-05-06T14:28:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f77fe700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb6e006c680 0x7fb6e006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f57fa700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] conn(0x7fb6e00723a0 0x7fb6e00747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:22.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 --> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6e0074e60 con 0x7fb6e00723a0 2024-05-06T14:28:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.368+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6e808dd00 con 0x7fb6f80ff390 2024-05-06T14:28:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.369+0000 7fb6fca43700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] conn(0x7fb6e00723a0 0x7fb6e00747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.369+0000 7fb6f77fe700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb6e006c680 0x7fb6e006eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8068dc0 tx=0x7fb6ec01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.369+0000 7fb6fca43700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] conn(0x7fb6e00723a0 0x7fb6e00747b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:22.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.370+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== osd.5 v2:172.21.15.190:6816/452463735 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb6e0074e60 con 0x7fb6e00723a0 2024-05-06T14:28:22.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.386+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 --> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb6f804ea50 con 0x7fb6e00723a0 2024-05-06T14:28:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.387+0000 7fb6f57fa700 1 -- 172.21.15.153:0/1298121861 <== osd.5 v2:172.21.15.190:6816/452463735 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb6f804ea50 con 0x7fb6e00723a0 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.387+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] conn(0x7fb6e00723a0 msgr2=0x7fb6e00747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.387+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] conn(0x7fb6e00723a0 0x7fb6e00747b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb6e006c680 msgr2=0x7fb6e006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb6e006c680 0x7fb6e006eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8068dc0 tx=0x7fb6ec01a040 comp rx=0 tx=0).stop 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 msgr2=0x7fb6f8194970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 0x7fb6f8194970 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800d900 tx=0x7fb6e800dcc0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 shutdown_connections 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:6816/452463735,v1:172.21.15.190:6817/452463735] conn(0x7fb6e00723a0 0x7fb6e00747b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6f80ff390 0x7fb6f8194970 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb6e006c680 0x7fb6e006eb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 --2- 172.21.15.153:0/1298121861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6f80ffd30 0x7fb6f8194eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 >> 172.21.15.153:0/1298121861 conn(0x7fb6f8074f50 msgr2=0x7fb6f80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 shutdown_connections 2024-05-06T14:28:22.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:22.388+0000 7fb6fe4a6700 1 -- 172.21.15.153:0/1298121861 wait complete. 2024-05-06T14:28:22.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:22 smithi190 ceph-mon[33671]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:22.995 INFO:teuthology.orchestra.run.smithi153.stdout:171798691849 2024-05-06T14:28:22.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-06T14:28:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.991+0000 7f192b484700 1 -- 172.21.15.153:0/3355380872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 msgr2=0x7f1924104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.991+0000 7f192b484700 1 --2- 172.21.15.153:0/3355380872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924104df0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1914009910 tx=0x7f1914009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.991+0000 7f192b484700 1 -- 172.21.15.153:0/3355380872 shutdown_connections 2024-05-06T14:28:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.991+0000 7f192b484700 1 --2- 172.21.15.153:0/3355380872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924104df0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.991+0000 7f192b484700 1 --2- 172.21.15.153:0/3355380872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1924068630 0x7f1924068a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.991+0000 7f192b484700 1 -- 172.21.15.153:0/3355380872 >> 172.21.15.153:0/3355380872 conn(0x7f1924074f50 msgr2=0x7f1924075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:23.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 -- 172.21.15.153:0/3355380872 shutdown_connections 2024-05-06T14:28:23.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 -- 172.21.15.153:0/3355380872 wait complete. 2024-05-06T14:28:23.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 Processor -- start 2024-05-06T14:28:23.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 -- start start 2024-05-06T14:28:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1924068630 0x7f1924195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1924196a40 con 0x7f1924068fd0 2024-05-06T14:28:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f192b484700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192419a7d0 con 0x7f1924068630 2024-05-06T14:28:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f1928a1f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.992+0000 7f1928a1f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55816/0 (socket says 172.21.15.153:55816) 2024-05-06T14:28:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f1928a1f700 1 -- 172.21.15.153:0/2879731788 learned_addr learned my addr 172.21.15.153:0/2879731788 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f1929220700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1924068630 0x7f1924195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f1928a1f700 1 -- 172.21.15.153:0/2879731788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1924068630 msgr2=0x7f1924195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f1928a1f700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1924068630 0x7f1924195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f1928a1f700 1 -- 172.21.15.153:0/2879731788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140095f0 con 0x7f1924068fd0 2024-05-06T14:28:23.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f1928a1f700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924196360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1914000c00 tx=0x7f191400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:23.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f191401d070 con 0x7f1924068fd0 2024-05-06T14:28:23.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1914005dc0 con 0x7f1924068fd0 2024-05-06T14:28:23.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192419aa50 con 0x7f1924068fd0 2024-05-06T14:28:23.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1914017400 con 0x7f1924068fd0 2024-05-06T14:28:23.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.993+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192419af40 con 0x7f1924068fd0 2024-05-06T14:28:23.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.994+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f192410d5d0 con 0x7f1924068fd0 2024-05-06T14:28:23.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f191400f460 con 0x7f1924068fd0 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f191a7fc700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f191006c680 0x7f191006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f191408cdf0 con 0x7f1924068fd0 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f1929220700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f191006c680 0x7f191006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f191a7fc700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] conn(0x7f1910072350 0x7f1910074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 --> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1910074e10 con 0x7f1910072350 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f191408d250 con 0x7f1924068fd0 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f1929a21700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] conn(0x7f1910072350 0x7f1910074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:23.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.995+0000 7f1929220700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f191006c680 0x7f191006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1920005950 tx=0x7f192000b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:23.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.996+0000 7f1929a21700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] conn(0x7f1910072350 0x7f1910074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:23.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:20.996+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== osd.2 v2:172.21.15.153:6818/3169611803 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1910074e10 con 0x7f1910072350 2024-05-06T14:28:23.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.012+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 --> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f192404ea50 con 0x7f1910072350 2024-05-06T14:28:23.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.012+0000 7f191a7fc700 1 -- 172.21.15.153:0/2879731788 <== osd.2 v2:172.21.15.153:6818/3169611803 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f192404ea50 con 0x7f1910072350 2024-05-06T14:28:23.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] conn(0x7f1910072350 msgr2=0x7f1910074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:23.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] conn(0x7f1910072350 0x7f1910074760 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f191006c680 msgr2=0x7f191006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f191006c680 0x7f191006eb30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1920005950 tx=0x7f192000b410 comp rx=0 tx=0).stop 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 msgr2=0x7f1924196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924196360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1914000c00 tx=0x7f191400bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 shutdown_connections 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6818/3169611803,v1:172.21.15.153:6819/3169611803] conn(0x7f1910072350 0x7f1910074760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f191006c680 0x7f191006eb30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1924068630 0x7f1924195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 --2- 172.21.15.153:0/2879731788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1924068fd0 0x7f1924196360 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:23.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 >> 172.21.15.153:0/2879731788 conn(0x7f1924074f50 msgr2=0x7f19240fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:23.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 shutdown_connections 2024-05-06T14:28:23.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:21.013+0000 7f192b484700 1 -- 172.21.15.153:0/2879731788 wait complete. 2024-05-06T14:28:23.777 INFO:teuthology.orchestra.run.smithi153.stdout:81604378648 2024-05-06T14:28:23.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-06T14:28:24.427 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:24 smithi153 ceph-mon[27177]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:24.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:24 smithi190 ceph-mon[33671]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:25.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:25.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 -- 172.21.15.153:0/3271324447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 msgr2=0x7f86f41016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 --2- 172.21.15.153:0/3271324447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f41016f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009910 tx=0x7f86f0009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 -- 172.21.15.153:0/3271324447 shutdown_connections 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 --2- 172.21.15.153:0/3271324447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86f4101c30 0x7f86f410a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 --2- 172.21.15.153:0/3271324447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f41016f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 -- 172.21.15.153:0/3271324447 >> 172.21.15.153:0/3271324447 conn(0x7f86f40fab80 msgr2=0x7f86f40fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.272+0000 7f86fb268700 1 -- 172.21.15.153:0/3271324447 shutdown_connections 2024-05-06T14:28:26.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.273+0000 7f86fb268700 1 -- 172.21.15.153:0/3271324447 wait complete. 2024-05-06T14:28:26.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86fb268700 1 Processor -- start 2024-05-06T14:28:26.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86fb268700 1 -- start start 2024-05-06T14:28:26.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86fb268700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f4195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86fb268700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86f4101c30 0x7f86f4196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86fb268700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f4196970 con 0x7f86f4101320 2024-05-06T14:28:26.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86fb268700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f419a700 con 0x7f86f4101c30 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f9004700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f4195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f9004700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f4195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55852/0 (socket says 172.21.15.153:55852) 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f9004700 1 -- 172.21.15.153:0/1101890905 learned_addr learned my addr 172.21.15.153:0/1101890905 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f8803700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86f4101c30 0x7f86f4196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f9004700 1 -- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86f4101c30 msgr2=0x7f86f4196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f9004700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86f4101c30 0x7f86f4196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.274+0000 7f86f9004700 1 -- 172.21.15.153:0/1101890905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f00095f0 con 0x7f86f4101320 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.275+0000 7f86f9004700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f4195d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f86f000fa00 tx=0x7f86f000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.275+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f001d070 con 0x7f86f4101320 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.275+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f419a980 con 0x7f86f4101320 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.275+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f86f000fe90 con 0x7f86f4101320 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.275+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f419ae70 con 0x7f86f4101320 2024-05-06T14:28:26.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.275+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f0022470 con 0x7f86f4101320 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.276+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86f410d530 con 0x7f86f4101320 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f86f0003680 con 0x7f86f4101320 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86ea7fc700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f86e006c680 0x7f86e006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f86f008c930 con 0x7f86f4101320 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86f8803700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f86e006c680 0x7f86e006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86ea7fc700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] conn(0x7f86e0072350 0x7f86e0074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 --> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86e0074e10 con 0x7f86e0072350 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.277+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86f00912a0 con 0x7f86f4101320 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.278+0000 7f86f9805700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] conn(0x7f86e0072350 0x7f86e0074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.278+0000 7f86f8803700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f86e006c680 0x7f86e006eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86f4197370 tx=0x7f86e4009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:26.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.278+0000 7f86f9805700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] conn(0x7f86e0072350 0x7f86e0074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:26.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.279+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== osd.4 v2:172.21.15.190:6808/1342651346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f86e0074e10 con 0x7f86e0072350 2024-05-06T14:28:26.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.295+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 --> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f86f404ea50 con 0x7f86e0072350 2024-05-06T14:28:26.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.295+0000 7f86ea7fc700 1 -- 172.21.15.153:0/1101890905 <== osd.4 v2:172.21.15.190:6808/1342651346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f86f404ea50 con 0x7f86e0072350 2024-05-06T14:28:26.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] conn(0x7f86e0072350 msgr2=0x7f86e0074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] conn(0x7f86e0072350 0x7f86e0074760 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f86e006c680 msgr2=0x7f86e006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f86e006c680 0x7f86e006eb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86f4197370 tx=0x7f86e4009260 comp rx=0 tx=0).stop 2024-05-06T14:28:26.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 msgr2=0x7f86f4195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f4195d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f86f000fa00 tx=0x7f86f000fa30 comp rx=0 tx=0).stop 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 shutdown_connections 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:6808/1342651346,v1:172.21.15.190:6809/1342651346] conn(0x7f86e0072350 0x7f86e0074760 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4101320 0x7f86f4195d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f86e006c680 0x7f86e006eb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 --2- 172.21.15.153:0/1101890905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86f4101c30 0x7f86f4196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 >> 172.21.15.153:0/1101890905 conn(0x7f86f40fab80 msgr2=0x7f86f4104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 shutdown_connections 2024-05-06T14:28:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.296+0000 7f86fb268700 1 -- 172.21.15.153:0/1101890905 wait complete. 2024-05-06T14:28:26.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:26 smithi153 ceph-mon[27177]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:26.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:26 smithi190 ceph-mon[33671]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:26.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 -- 172.21.15.153:0/1248060449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ffd30 msgr2=0x7fc688103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 --2- 172.21.15.153:0/1248060449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ffd30 0x7fc688103ce0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc674009910 tx=0x7fc674009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 -- 172.21.15.153:0/1248060449 shutdown_connections 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 --2- 172.21.15.153:0/1248060449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ffd30 0x7fc688103ce0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 --2- 172.21.15.153:0/1248060449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6880ff390 0x7fc6880ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 -- 172.21.15.153:0/1248060449 >> 172.21.15.153:0/1248060449 conn(0x7fc688074f50 msgr2=0x7fc688075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 -- 172.21.15.153:0/1248060449 shutdown_connections 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.469+0000 7fc68eb05700 1 -- 172.21.15.153:0/1248060449 wait complete. 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68eb05700 1 Processor -- start 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68eb05700 1 -- start start 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68eb05700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 0x7fc688100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68eb05700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6880ffd30 0x7fc6881012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68eb05700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688101890 con 0x7fc6880ff390 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68eb05700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688101a00 con 0x7fc6880ffd30 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68c8a1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 0x7fc688100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68c8a1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 0x7fc688100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55870/0 (socket says 172.21.15.153:55870) 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc68c8a1700 1 -- 172.21.15.153:0/2902201750 learned_addr learned my addr 172.21.15.153:0/2902201750 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.470+0000 7fc687fff700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6880ffd30 0x7fc6881012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc68c8a1700 1 -- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6880ffd30 msgr2=0x7fc6881012c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc68c8a1700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6880ffd30 0x7fc6881012c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc68c8a1700 1 -- 172.21.15.153:0/2902201750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6740095f0 con 0x7fc6880ff390 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc68c8a1700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 0x7fc688100d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00b4b0 tx=0x7fc67c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc67c011800 con 0x7fc6880ff390 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6881a2020 con 0x7fc6880ff390 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc67c011e10 con 0x7fc6880ff390 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc67c00f2e0 con 0x7fc6880ff390 2024-05-06T14:28:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.471+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6881a2430 con 0x7fc6880ff390 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.472+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc68810c560 con 0x7fc6880ff390 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fc67c010380 con 0x7fc6880ff390 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc685ffb700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc67806c680 0x7fc67806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc67c08aff0 con 0x7fc6880ff390 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc687fff700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc67806c680 0x7fc67806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc685ffb700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] conn(0x7fc678072350 0x7fc678074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 --> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc678074e10 con 0x7fc678072350 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc67c08d6c0 con 0x7fc6880ff390 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.473+0000 7fc68d0a2700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] conn(0x7fc678072350 0x7fc678074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.474+0000 7fc687fff700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc67806c680 0x7fc67806eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6881028f0 tx=0x7fc67401a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.474+0000 7fc68d0a2700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] conn(0x7fc678072350 0x7fc678074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.475+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== osd.3 v2:172.21.15.190:6800/492034660 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc678074e10 con 0x7fc678072350 2024-05-06T14:28:26.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.490+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 --> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc68804ea50 con 0x7fc678072350 2024-05-06T14:28:26.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc685ffb700 1 -- 172.21.15.153:0/2902201750 <== osd.3 v2:172.21.15.190:6800/492034660 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc68804ea50 con 0x7fc678072350 2024-05-06T14:28:26.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] conn(0x7fc678072350 msgr2=0x7fc678074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] conn(0x7fc678072350 0x7fc678074760 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc67806c680 msgr2=0x7fc67806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc67806c680 0x7fc67806eb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6881028f0 tx=0x7fc67401a040 comp rx=0 tx=0).stop 2024-05-06T14:28:26.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 msgr2=0x7fc688100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 0x7fc688100d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00b4b0 tx=0x7fc67c00b7c0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 shutdown_connections 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:6800/492034660,v1:172.21.15.190:6801/492034660] conn(0x7fc678072350 0x7fc678074760 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6880ff390 0x7fc688100d80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc67806c680 0x7fc67806eb30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 --2- 172.21.15.153:0/2902201750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6880ffd30 0x7fc6881012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 >> 172.21.15.153:0/2902201750 conn(0x7fc688074f50 msgr2=0x7fc6880686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:26.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.492+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 shutdown_connections 2024-05-06T14:28:26.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:26.493+0000 7fc68eb05700 1 -- 172.21.15.153:0/2902201750 wait complete. 2024-05-06T14:28:27.195 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:27.207 INFO:teuthology.orchestra.run.smithi153.stdout:150323855375 2024-05-06T14:28:27.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-06T14:28:27.603 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:28.103 INFO:teuthology.orchestra.run.smithi153.stdout:115964117010 2024-05-06T14:28:28.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-06T14:28:28.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 -- 172.21.15.153:0/4036271298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 msgr2=0x7f99500ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 --2- 172.21.15.153:0/4036271298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f99500ff6b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9940009910 tx=0x7f9940009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 -- 172.21.15.153:0/4036271298 shutdown_connections 2024-05-06T14:28:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 --2- 172.21.15.153:0/4036271298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 0x7f9950103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 --2- 172.21.15.153:0/4036271298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f99500ff6b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 -- 172.21.15.153:0/4036271298 >> 172.21.15.153:0/4036271298 conn(0x7f99500fab80 msgr2=0x7f99500fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:28.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:28 smithi153 ceph-mon[27177]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 -- 172.21.15.153:0/4036271298 shutdown_connections 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.318+0000 7f9956056700 1 -- 172.21.15.153:0/4036271298 wait complete. 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.319+0000 7f9956056700 1 Processor -- start 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.319+0000 7f9956056700 1 -- start start 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.319+0000 7f9956056700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f9950100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.319+0000 7f9956056700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 0x7f9950101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:28.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.319+0000 7f9956056700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9950101770 con 0x7f99500ff2e0 2024-05-06T14:28:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.319+0000 7f9956056700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99501018b0 con 0x7f99500ffc80 2024-05-06T14:28:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994effd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 0x7f9950101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f9950100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994effd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 0x7f9950101230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:41306/0 (socket says 172.21.15.153:41306) 2024-05-06T14:28:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f9950100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55890/0 (socket says 172.21.15.153:55890) 2024-05-06T14:28:28.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994effd700 1 -- 172.21.15.153:0/45206278 learned_addr learned my addr 172.21.15.153:0/45206278 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994f7fe700 1 -- 172.21.15.153:0/45206278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 msgr2=0x7f9950101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994f7fe700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 0x7f9950101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994f7fe700 1 -- 172.21.15.153:0/45206278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400095f0 con 0x7f99500ff2e0 2024-05-06T14:28:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994f7fe700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f9950100cf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9940009910 tx=0x7f994000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994001d070 con 0x7f99500ff2e0 2024-05-06T14:28:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f994000f460 con 0x7f99500ff2e0 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9940017630 con 0x7f99500ff2e0 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.320+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9950101b30 con 0x7f99500ff2e0 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.321+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99501a2280 con 0x7f99500ff2e0 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.321+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f995010c4c0 con 0x7f99500ff2e0 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f994000f9e0 con 0x7f99500ff2e0 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994cff9700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f993806c680 0x7f993806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:28.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f994008c830 con 0x7f99500ff2e0 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994effd700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f993806c680 0x7f993806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994cff9700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] conn(0x7f9938072350 0x7f9938074760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 --> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9938074e10 con 0x7f9938072350 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99400912a0 con 0x7f99500ff2e0 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.323+0000 7f994ffff700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] conn(0x7f9938072350 0x7f9938074760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.324+0000 7f994effd700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f993806c680 0x7f993806eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f99501024e0 tx=0x7f994400a3b0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.324+0000 7f994ffff700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] conn(0x7f9938072350 0x7f9938074760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.325+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== osd.1 v2:172.21.15.153:6810/3716718209 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9938074e10 con 0x7f9938072350 2024-05-06T14:28:28.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.346+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 --> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f995004ea50 con 0x7f9938072350 2024-05-06T14:28:28.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.346+0000 7f994cff9700 1 -- 172.21.15.153:0/45206278 <== osd.1 v2:172.21.15.153:6810/3716718209 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f995004ea50 con 0x7f9938072350 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] conn(0x7f9938072350 msgr2=0x7f9938074760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] conn(0x7f9938072350 0x7f9938074760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f993806c680 msgr2=0x7f993806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f993806c680 0x7f993806eb30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f99501024e0 tx=0x7f994400a3b0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 msgr2=0x7f9950100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f9950100cf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9940009910 tx=0x7f994000bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 shutdown_connections 2024-05-06T14:28:28.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99500ff2e0 0x7f9950100cf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6810/3716718209,v1:172.21.15.153:6811/3716718209] conn(0x7f9938072350 0x7f9938074760 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f993806c680 0x7f993806eb30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 --2- 172.21.15.153:0/45206278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99500ffc80 0x7f9950101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:28.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 >> 172.21.15.153:0/45206278 conn(0x7f99500fab80 msgr2=0x7f9950069070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:28.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 shutdown_connections 2024-05-06T14:28:28.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:28.347+0000 7f9956056700 1 -- 172.21.15.153:0/45206278 wait complete. 2024-05-06T14:28:28.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:28 smithi190 ceph-mon[33671]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:29.033 INFO:teuthology.orchestra.run.smithi153.stdout:60129542174 2024-05-06T14:28:29.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-06T14:28:29.965 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:30.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:30 smithi153 ceph-mon[27177]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:30.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:30 smithi190 ceph-mon[33671]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:30.442 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:32.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:32 smithi153 ceph-mon[27177]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:32.364 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:32.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:32 smithi190 ceph-mon[33671]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:33.107 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:34.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:34 smithi153 ceph-mon[27177]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:34.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:34 smithi190 ceph-mon[33671]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 -- 172.21.15.153:0/2277927757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ff390 msgr2=0x7ff8c80ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/2277927757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ff390 0x7ff8c80ff760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009910 tx=0x7ff8b8009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 -- 172.21.15.153:0/2277927757 shutdown_connections 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/2277927757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/2277927757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ff390 0x7ff8c80ff760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 -- 172.21.15.153:0/2277927757 >> 172.21.15.153:0/2277927757 conn(0x7ff8c8074f50 msgr2=0x7ff8c8075350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 -- 172.21.15.153:0/2277927757 shutdown_connections 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.504+0000 7ff8cfa18700 1 -- 172.21.15.153:0/2277927757 wait complete. 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cfa18700 1 Processor -- start 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cfa18700 1 -- start start 2024-05-06T14:28:34.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cfa18700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8c80ff390 0x7ff8c8194d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cfa18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8195290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cfa18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c8195970 con 0x7ff8c80ffd30 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cfa18700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c8199700 con 0x7ff8c80ff390 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8ccfb3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8195290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8ccfb3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8195290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50918/0 (socket says 172.21.15.153:50918) 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8ccfb3700 1 -- 172.21.15.153:0/1985962158 learned_addr learned my addr 172.21.15.153:0/1985962158 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8ccfb3700 1 -- 172.21.15.153:0/1985962158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8c80ff390 msgr2=0x7ff8c8194d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8cd7b4700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8c80ff390 0x7ff8c8194d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:34.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8ccfb3700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8c80ff390 0x7ff8c8194d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.505+0000 7ff8ccfb3700 1 -- 172.21.15.153:0/1985962158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80095f0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.506+0000 7ff8ccfb3700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8195290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400b480 tx=0x7ff8c400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.506+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c40118c0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.506+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c81999e0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.506+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff8c4010380 con 0x7ff8c80ffd30 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.506+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c400f4f0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.506+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c8199f30 con 0x7ff8c80ffd30 2024-05-06T14:28:34.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.507+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c8107650 con 0x7ff8c80ffd30 2024-05-06T14:28:34.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.507+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff8c40104f0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.508+0000 7ff8be7fc700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff8b406c680 0x7ff8b406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:34.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.508+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff8c408b180 con 0x7ff8c80ffd30 2024-05-06T14:28:34.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.508+0000 7ff8cd7b4700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff8b406c680 0x7ff8b406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:34.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.508+0000 7ff8cd7b4700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff8b406c680 0x7ff8b406eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800b5c0 tx=0x7ff8b8005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:34.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.512+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff8c4055ba0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.707+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff8c819a1b0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.708+0000 7ff8be7fc700 1 -- 172.21.15.153:0/1985962158 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff8c40591c0 con 0x7ff8c80ffd30 2024-05-06T14:28:34.696 INFO:teuthology.orchestra.run.smithi153.stdout:171798691851 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff8b406c680 msgr2=0x7ff8b406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff8b406c680 0x7ff8b406eb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800b5c0 tx=0x7ff8b8005960 comp rx=0 tx=0).stop 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 msgr2=0x7ff8c8195290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8195290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400b480 tx=0x7ff8c400b840 comp rx=0 tx=0).stop 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 shutdown_connections 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff8b406c680 0x7ff8b406eb30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8c80ff390 0x7ff8c8194d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:34.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 --2- 172.21.15.153:0/1985962158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8c80ffd30 0x7ff8c8195290 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:34.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 >> 172.21.15.153:0/1985962158 conn(0x7ff8c8074f50 msgr2=0x7ff8c80fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:34.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 shutdown_connections 2024-05-06T14:28:34.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:34.710+0000 7ff8cfa18700 1 -- 172.21.15.153:0/1985962158 wait complete. 2024-05-06T14:28:35.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:35 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1985962158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T14:28:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:35 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1985962158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T14:28:35.570 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691851 for osd.5 2024-05-06T14:28:35.570 DEBUG:teuthology.parallel:result is None 2024-05-06T14:28:36.209 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:36 smithi153 ceph-mon[27177]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:36.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:36 smithi190 ceph-mon[33671]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:36.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/4033569134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 msgr2=0x7fb4f40ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:36.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/4033569134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f40ff690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009910 tx=0x7fb4ec009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/4033569134 shutdown_connections 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/4033569134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f40ffc60 0x7fb4f4103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/4033569134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f40ff690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/4033569134 >> 172.21.15.153:0/4033569134 conn(0x7fb4f40fab80 msgr2=0x7fb4f40fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/4033569134 shutdown_connections 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/4033569134 wait complete. 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.893+0000 7fb4f8a8d700 1 Processor -- start 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f8a8d700 1 -- start start 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f8a8d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f4196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f8a8d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f40ffc60 0x7fb4f41970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f8a8d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f41976f0 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f8a8d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4197830 con 0x7fb4f40ffc60 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f4196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f4196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50938/0 (socket says 172.21.15.153:50938) 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 -- 172.21.15.153:0/149443106 learned_addr learned my addr 172.21.15.153:0/149443106 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f1d9b700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f40ffc60 0x7fb4f41970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 -- 172.21.15.153:0/149443106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f40ffc60 msgr2=0x7fb4f41970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f40ffc60 0x7fb4f41970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 -- 172.21.15.153:0/149443106 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec0095f0 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f259c700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f4196b90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec006010 tx=0x7fb4ec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4ec01d070 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb4ec00f460 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.894+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f419fc00 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.895+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4ec017630 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.895+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f41a0010 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.895+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f41075d0 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.896+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fb4ec00f5d0 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.896+0000 7fb4e37fe700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb4dc06c680 0x7fb4dc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.897+0000 7fb4f1d9b700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb4dc06c680 0x7fb4dc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.897+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb4ec08c950 con 0x7fb4f40ff2c0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.897+0000 7fb4f1d9b700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb4dc06c680 0x7fb4dc06eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4069510 tx=0x7fb4e800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:36.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.901+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb4ec057370 con 0x7fb4f40ff2c0 2024-05-06T14:28:37.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 -- 172.21.15.153:0/1534456443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 msgr2=0x7f622410a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:37.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1534456443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622410a180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6218009910 tx=0x7f6218009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:37.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 -- 172.21.15.153:0/1534456443 shutdown_connections 2024-05-06T14:28:37.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1534456443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622410a180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1534456443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6224101370 0x7f6224101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 -- 172.21.15.153:0/1534456443 >> 172.21.15.153:0/1534456443 conn(0x7f62240fabd0 msgr2=0x7f62240fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 -- 172.21.15.153:0/1534456443 shutdown_connections 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.804+0000 7f622a1b3700 1 -- 172.21.15.153:0/1534456443 wait complete. 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.805+0000 7f622a1b3700 1 Processor -- start 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.805+0000 7f622a1b3700 1 -- start start 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f622a1b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6224101370 0x7f622419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f622a1b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f622a1b3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622419c690 con 0x7f6224101c80 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f622a1b3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224195b40 con 0x7f6224101370 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:37.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622419c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50926/0 (socket says 172.21.15.153:50926) 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stdout:81604378650 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 -- 172.21.15.153:0/1445642625 learned_addr learned my addr 172.21.15.153:0/1445642625 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f62237fe700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6224101370 0x7f622419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 -- 172.21.15.153:0/1445642625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6224101370 msgr2=0x7f622419bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6224101370 0x7f622419bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 -- 172.21.15.153:0/1445642625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180095f0 con 0x7f6224101c80 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.806+0000 7f6222ffd700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622419c000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6218006010 tx=0x7f621800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.807+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f621801d070 con 0x7f6224101c80 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.807+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6224195dc0 con 0x7f6224101c80 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.807+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f621800f460 con 0x7f6224101c80 2024-05-06T14:28:37.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.807+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62241962b0 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.807+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6218017630 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.808+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62241077f0 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.808+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f621800f5d0 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.809+0000 7f6220ff9700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f620c06c680 0x7f620c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.809+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f621808c930 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.809+0000 7f62237fe700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f620c06c680 0x7f620c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.810+0000 7f62237fe700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f620c06c680 0x7f620c06eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6214005950 tx=0x7f621400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:36.813+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f621805adc0 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.014+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f622404ea50 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.015+0000 7f6220ff9700 1 -- 172.21.15.153:0/1445642625 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f621805a950 con 0x7f6224101c80 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f620c06c680 msgr2=0x7f620c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f620c06c680 0x7f620c06eb30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6214005950 tx=0x7f621400b410 comp rx=0 tx=0).stop 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 msgr2=0x7f622419c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:37.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622419c000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6218006010 tx=0x7f621800bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 shutdown_connections 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f620c06c680 0x7f620c06eb30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6224101370 0x7f622419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 --2- 172.21.15.153:0/1445642625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6224101c80 0x7f622419c000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 >> 172.21.15.153:0/1445642625 conn(0x7f62240fabd0 msgr2=0x7f6224104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 shutdown_connections 2024-05-06T14:28:37.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.017+0000 7f622a1b3700 1 -- 172.21.15.153:0/1445642625 wait complete. 2024-05-06T14:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.100+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb4f404ea50 con 0x7fb4f40ff2c0 2024-05-06T14:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.100+0000 7fb4e37fe700 1 -- 172.21.15.153:0/149443106 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb4ec05a990 con 0x7fb4f40ff2c0 2024-05-06T14:28:37.089 INFO:teuthology.orchestra.run.smithi153.stdout:38654705702 2024-05-06T14:28:37.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.102+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb4dc06c680 msgr2=0x7fb4dc06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.102+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb4dc06c680 0x7fb4dc06eb30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4069510 tx=0x7fb4e800b410 comp rx=0 tx=0).stop 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.102+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 msgr2=0x7fb4f4196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.102+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f4196b90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec006010 tx=0x7fb4ec00bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 shutdown_connections 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f40ff2c0 0x7fb4f4196b90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fb4dc06c680 0x7fb4dc06eb30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 --2- 172.21.15.153:0/149443106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f40ffc60 0x7fb4f41970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 >> 172.21.15.153:0/149443106 conn(0x7fb4f40fab80 msgr2=0x7fb4f40fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 shutdown_connections 2024-05-06T14:28:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:37.103+0000 7fb4f8a8d700 1 -- 172.21.15.153:0/149443106 wait complete. 2024-05-06T14:28:37.102 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:37 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1445642625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T14:28:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:37 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1445642625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T14:28:37.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-05-06T14:28:37.877 DEBUG:teuthology.parallel:result is None 2024-05-06T14:28:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:38 smithi153 ceph-mon[27177]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:38 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/149443106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T14:28:38.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:38 smithi190 ceph-mon[33671]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:38.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:38 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/149443106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T14:28:38.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2024-05-06T14:28:38.781 DEBUG:teuthology.parallel:result is None 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 -- 172.21.15.153:0/945243782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ffb30 msgr2=0x7f909810a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 --2- 172.21.15.153:0/945243782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ffb30 0x7f909810a5a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9084009910 tx=0x7f9084009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 -- 172.21.15.153:0/945243782 shutdown_connections 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 --2- 172.21.15.153:0/945243782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ffb30 0x7f909810a5a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 --2- 172.21.15.153:0/945243782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ff220 0x7f90980ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 -- 172.21.15.153:0/945243782 >> 172.21.15.153:0/945243782 conn(0x7f90980747e0 msgr2=0x7f9098074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 -- 172.21.15.153:0/945243782 shutdown_connections 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 -- 172.21.15.153:0/945243782 wait complete. 2024-05-06T14:28:39.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 Processor -- start 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.015+0000 7f909f1b7700 1 -- start start 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909f1b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 0x7f909819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909f1b7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ffb30 0x7f909819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909f1b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909819c5c0 con 0x7f90980ff220 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909f1b7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098195a70 con 0x7f90980ffb30 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 0x7f909819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 0x7f909819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50952/0 (socket says 172.21.15.153:50952) 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 -- 172.21.15.153:0/1493266570 learned_addr learned my addr 172.21.15.153:0/1493266570 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f9097fff700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ffb30 0x7f909819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 -- 172.21.15.153:0/1493266570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ffb30 msgr2=0x7f909819bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ffb30 0x7f909819bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 -- 172.21.15.153:0/1493266570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90840095f0 con 0x7f90980ff220 2024-05-06T14:28:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.016+0000 7f909cf53700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 0x7f909819b9f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c8a0 tx=0x7f908c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:39.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.017+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f908c0078e0 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.017+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f908c00f450 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.017+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9098195d50 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.017+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f908c00e5e0 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.017+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9098196320 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.018+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9098107ca0 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.018+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f908c007a40 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.019+0000 7f9095ffb700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f908806c680 0x7f908806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.019+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f908c08b3e0 con 0x7f90980ff220 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.019+0000 7f9097fff700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f908806c680 0x7f908806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:39.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.019+0000 7f9097fff700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f908806c680 0x7f908806eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90980fddf0 tx=0x7f908400b770 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:39.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.023+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f908c059910 con 0x7f90980ff220 2024-05-06T14:28:39.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.220+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f909804ea50 con 0x7f90980ff220 2024-05-06T14:28:39.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.220+0000 7f9095ffb700 1 -- 172.21.15.153:0/1493266570 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f908c0594a0 con 0x7f90980ff220 2024-05-06T14:28:39.209 INFO:teuthology.orchestra.run.smithi153.stdout:60129542176 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f908806c680 msgr2=0x7f908806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f908806c680 0x7f908806eb30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90980fddf0 tx=0x7f908400b770 comp rx=0 tx=0).stop 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 msgr2=0x7f909819b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 0x7f909819b9f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c8a0 tx=0x7f908c00cbb0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 shutdown_connections 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980ff220 0x7f909819b9f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f908806c680 0x7f908806eb30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 --2- 172.21.15.153:0/1493266570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90980ffb30 0x7f909819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 >> 172.21.15.153:0/1493266570 conn(0x7f90980747e0 msgr2=0x7f9098104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 shutdown_connections 2024-05-06T14:28:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.223+0000 7f909f1b7700 1 -- 172.21.15.153:0/1493266570 wait complete. 2024-05-06T14:28:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 -- 172.21.15.153:0/3899135056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ff2c0 msgr2=0x7f333c0ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 --2- 172.21.15.153:0/3899135056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ff2c0 0x7f333c0ff690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f333400b0a0 tx=0x7f333400b3b0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 -- 172.21.15.153:0/3899135056 shutdown_connections 2024-05-06T14:28:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 --2- 172.21.15.153:0/3899135056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f333c0ffc60 0x7f333c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 --2- 172.21.15.153:0/3899135056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ff2c0 0x7f333c0ff690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 -- 172.21.15.153:0/3899135056 >> 172.21.15.153:0/3899135056 conn(0x7f333c0fab80 msgr2=0x7f333c0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 -- 172.21.15.153:0/3899135056 shutdown_connections 2024-05-06T14:28:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.714+0000 7f334113f700 1 -- 172.21.15.153:0/3899135056 wait complete. 2024-05-06T14:28:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f334113f700 1 Processor -- start 2024-05-06T14:28:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f334113f700 1 -- start start 2024-05-06T14:28:39.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f334113f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f333c0ff2c0 0x7f333c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:39.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f334113f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 0x7f333c1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:39.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f334113f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1969c0 con 0x7f333c0ffc60 2024-05-06T14:28:39.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f334113f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c19a750 con 0x7f333c0ff2c0 2024-05-06T14:28:39.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f333a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 0x7f333c1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:39.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f333a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 0x7f333c1962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55674/0 (socket says 172.21.15.153:55674) 2024-05-06T14:28:39.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f333a59c700 1 -- 172.21.15.153:0/533451342 learned_addr learned my addr 172.21.15.153:0/533451342 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:39.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.715+0000 7f333ad9d700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f333c0ff2c0 0x7f333c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:39.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f333a59c700 1 -- 172.21.15.153:0/533451342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f333c0ff2c0 msgr2=0x7f333c195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f333a59c700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f333c0ff2c0 0x7f333c195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f333a59c700 1 -- 172.21.15.153:0/533451342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334009d00 con 0x7f333c0ffc60 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f333a59c700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 0x7f333c1962e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f332400d900 tx=0x7f332400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3324009900 con 0x7f333c0ffc60 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c19aa30 con 0x7f333c0ffc60 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3324010460 con 0x7f333c0ffc60 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f332400f5f0 con 0x7f333c0ffc60 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.716+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333c19af80 con 0x7f333c0ffc60 2024-05-06T14:28:39.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.717+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333c04ea50 con 0x7f333c0ffc60 2024-05-06T14:28:39.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.717+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f33240105d0 con 0x7f333c0ffc60 2024-05-06T14:28:39.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.718+0000 7f3333fff700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f332806c680 0x7f332806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:39.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.718+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f332408b010 con 0x7f333c0ffc60 2024-05-06T14:28:39.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.718+0000 7f333ad9d700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f332806c680 0x7f332806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:39.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.718+0000 7f333ad9d700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f332806c680 0x7f332806eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3334000f80 tx=0x7f33340060c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:39.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.723+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3324055ab0 con 0x7f333c0ffc60 2024-05-06T14:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.913+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f333c197100 con 0x7f333c0ffc60 2024-05-06T14:28:39.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.914+0000 7f3333fff700 1 -- 172.21.15.153:0/533451342 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f33240590d0 con 0x7f333c0ffc60 2024-05-06T14:28:39.902 INFO:teuthology.orchestra.run.smithi153.stdout:150323855377 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f332806c680 msgr2=0x7f332806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f332806c680 0x7f332806eb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3334000f80 tx=0x7f33340060c0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 msgr2=0x7f333c1962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 0x7f333c1962e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f332400d900 tx=0x7f332400dc10 comp rx=0 tx=0).stop 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 shutdown_connections 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f332806c680 0x7f332806eb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f333c0ff2c0 0x7f333c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 --2- 172.21.15.153:0/533451342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333c0ffc60 0x7f333c1962e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 >> 172.21.15.153:0/533451342 conn(0x7f333c0fab80 msgr2=0x7f333c068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 shutdown_connections 2024-05-06T14:28:39.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:39.916+0000 7f334113f700 1 -- 172.21.15.153:0/533451342 wait complete. 2024-05-06T14:28:40.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:40 smithi153 ceph-mon[27177]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:40.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:40 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1493266570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T14:28:40.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:40 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/533451342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T14:28:40.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:40.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 -- 172.21.15.153:0/3959933147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 msgr2=0x7f7a0c10a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 --2- 172.21.15.153:0/3959933147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c10a610 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009910 tx=0x7f7a08009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 -- 172.21.15.153:0/3959933147 shutdown_connections 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 --2- 172.21.15.153:0/3959933147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c10a610 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 --2- 172.21.15.153:0/3959933147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a0c0ff290 0x7f7a0c0ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 -- 172.21.15.153:0/3959933147 >> 172.21.15.153:0/3959933147 conn(0x7f7a0c074850 msgr2=0x7f7a0c074c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 -- 172.21.15.153:0/3959933147 shutdown_connections 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.367+0000 7f7a13d63700 1 -- 172.21.15.153:0/3959933147 wait complete. 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a13d63700 1 Processor -- start 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a13d63700 1 -- start start 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a13d63700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a0c0ff290 0x7f7a0c195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:40.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a13d63700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c1962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a13d63700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c196980 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a13d63700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c19a710 con 0x7f7a0c0ff290 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a112fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c1962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a112fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c1962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55684/0 (socket says 172.21.15.153:55684) 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a112fe700 1 -- 172.21.15.153:0/783583069 learned_addr learned my addr 172.21.15.153:0/783583069 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.368+0000 7f7a11aff700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a0c0ff290 0x7f7a0c195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a112fe700 1 -- 172.21.15.153:0/783583069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a0c0ff290 msgr2=0x7f7a0c195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a112fe700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a0c0ff290 0x7f7a0c195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a112fe700 1 -- 172.21.15.153:0/783583069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080095f0 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a112fe700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c1962a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a08006010 tx=0x7f7a0800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a0801d070 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7a0800f460 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a08017630 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c19a990 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.369+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c19ae30 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.370+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0c107d10 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.370+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7a0800f5d0 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.371+0000 7f7a02ffd700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f79f806c680 0x7f79f806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.371+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7a0808c9a0 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.371+0000 7f7a11aff700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f79f806c680 0x7f79f806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.371+0000 7f7a11aff700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f79f806c680 0x7f79f806eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c06a5c0 tx=0x7f79fc00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:40.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.375+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7a080573c0 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:40 smithi190 ceph-mon[33671]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:40.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:40 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1493266570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T14:28:40.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:40 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/533451342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T14:28:40.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:40.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.574+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7a0c02cc30 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.574+0000 7f7a02ffd700 1 -- 172.21.15.153:0/783583069 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7a0805a9e0 con 0x7f7a0c0ffba0 2024-05-06T14:28:40.563 INFO:teuthology.orchestra.run.smithi153.stdout:115964117013 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f79f806c680 msgr2=0x7f79f806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f79f806c680 0x7f79f806eb30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c06a5c0 tx=0x7f79fc00b410 comp rx=0 tx=0).stop 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 msgr2=0x7f7a0c1962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c1962a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7a08006010 tx=0x7f7a0800bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 shutdown_connections 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f79f806c680 0x7f79f806eb30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a0c0ff290 0x7f7a0c195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 --2- 172.21.15.153:0/783583069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a0c0ffba0 0x7f7a0c1962a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.576+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 >> 172.21.15.153:0/783583069 conn(0x7f7a0c074850 msgr2=0x7f7a0c104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.577+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 shutdown_connections 2024-05-06T14:28:40.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:40.577+0000 7f7a13d63700 1 -- 172.21.15.153:0/783583069 wait complete. 2024-05-06T14:28:40.880 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-05-06T14:28:40.880 DEBUG:teuthology.parallel:result is None 2024-05-06T14:28:41.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:41 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/783583069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T14:28:41.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:41 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/783583069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T14:28:41.650 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855377 for osd.4 2024-05-06T14:28:41.650 DEBUG:teuthology.parallel:result is None 2024-05-06T14:28:42.072 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117010 got 115964117013 for osd.3 2024-05-06T14:28:42.072 DEBUG:teuthology.parallel:result is None 2024-05-06T14:28:42.073 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-06T14:28:42.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-06T14:28:42.282 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:42.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:42 smithi153 ceph-mon[27177]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:42.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:42 smithi190 ceph-mon[33671]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:43.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 -- 172.21.15.153:0/2689367822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ff2c0 msgr2=0x7f09780ff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 --2- 172.21.15.153:0/2689367822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ff2c0 0x7f09780ff690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f097400b0a0 tx=0x7f097400b3b0 comp rx=0 tx=0).stop 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 -- 172.21.15.153:0/2689367822 shutdown_connections 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 --2- 172.21.15.153:0/2689367822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09780ffc60 0x7f0978103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 --2- 172.21.15.153:0/2689367822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ff2c0 0x7f09780ff690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 -- 172.21.15.153:0/2689367822 >> 172.21.15.153:0/2689367822 conn(0x7f09780fab80 msgr2=0x7f09780fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 -- 172.21.15.153:0/2689367822 shutdown_connections 2024-05-06T14:28:43.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.904+0000 7f097ebc9700 1 -- 172.21.15.153:0/2689367822 wait complete. 2024-05-06T14:28:43.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f097ebc9700 1 Processor -- start 2024-05-06T14:28:43.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f097ebc9700 1 -- start start 2024-05-06T14:28:43.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f097ebc9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09780ff2c0 0x7f0978194d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:43.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f097ebc9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 0x7f0978195250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:43.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f097ebc9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978195930 con 0x7f09780ffc60 2024-05-06T14:28:43.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f097ebc9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09781996c0 con 0x7f09780ff2c0 2024-05-06T14:28:43.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f096ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 0x7f0978195250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:43.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f096ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 0x7f0978195250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55690/0 (socket says 172.21.15.153:55690) 2024-05-06T14:28:43.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.905+0000 7f096ffff700 1 -- 172.21.15.153:0/3402580646 learned_addr learned my addr 172.21.15.153:0/3402580646 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:43.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f097c965700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09780ff2c0 0x7f0978194d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:43.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096ffff700 1 -- 172.21.15.153:0/3402580646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09780ff2c0 msgr2=0x7f0978194d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:43.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096ffff700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09780ff2c0 0x7f0978194d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:43.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096ffff700 1 -- 172.21.15.153:0/3402580646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974009d00 con 0x7f09780ffc60 2024-05-06T14:28:43.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096ffff700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 0x7f0978195250 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f096800b480 tx=0x7f096800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:43.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09680118c0 con 0x7f09780ffc60 2024-05-06T14:28:43.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0968010380 con 0x7f09780ffc60 2024-05-06T14:28:43.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09781999a0 con 0x7f09780ffc60 2024-05-06T14:28:43.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f096800f4f0 con 0x7f09780ffc60 2024-05-06T14:28:43.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.906+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0978199ef0 con 0x7f09780ffc60 2024-05-06T14:28:43.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.907+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09781075d0 con 0x7f09780ffc60 2024-05-06T14:28:43.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.908+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f09680104f0 con 0x7f09780ffc60 2024-05-06T14:28:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.908+0000 7f096dffb700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f096006c680 0x7f096006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.908+0000 7f097c965700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f096006c680 0x7f096006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.908+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f096808b0c0 con 0x7f09780ffc60 2024-05-06T14:28:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.908+0000 7f097c965700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f096006c680 0x7f096006eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f097400b830 tx=0x7f09740097c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:43.912+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0968055b60 con 0x7f09780ffc60 2024-05-06T14:28:44.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.099+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f097819a2b0 con 0x7f096006c680 2024-05-06T14:28:44.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.100+0000 7f096dffb700 1 -- 172.21.15.153:0/3402580646 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28111 (secure 0 0 0) 0x7f097819a2b0 con 0x7f096006c680 2024-05-06T14:28:44.087 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f096006c680 msgr2=0x7f096006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f096006c680 0x7f096006eb30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f097400b830 tx=0x7f09740097c0 comp rx=0 tx=0).stop 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 msgr2=0x7f0978195250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 0x7f0978195250 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f096800b480 tx=0x7f096800b840 comp rx=0 tx=0).stop 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 shutdown_connections 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f096006c680 0x7f096006eb30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09780ff2c0 0x7f0978194d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:44.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 --2- 172.21.15.153:0/3402580646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09780ffc60 0x7f0978195250 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:44.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 >> 172.21.15.153:0/3402580646 conn(0x7f09780fab80 msgr2=0x7f09780fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:44.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 shutdown_connections 2024-05-06T14:28:44.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:44.102+0000 7f097ebc9700 1 -- 172.21.15.153:0/3402580646 wait complete. 2024-05-06T14:28:44.091 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-05-06T14:28:44.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:44 smithi153 ceph-mon[27177]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:44.438 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:44 smithi190 ceph-mon[33671]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:44.783 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-05-06T14:28:42.970349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002234"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-06T14:27:54.590081+0000","last_change":"2024-05-06T14:27:28.077420+0000","last_active":"2024-05-06T14:27:54.590081+0000","last_peered":"2024-05-06T14:27:54.590081+0000","last_clean":"2024-05-06T14:27:54.590081+0000","last_became_active":"2024-05-06T14:27:28.077132+0000","last_became_peered":"2024-05-06T14:27:28.077132+0000","last_unstale":"2024-05-06T14:27:54.590081+0000","last_undegraded":"2024-05-06T14:27:54.590081+0000","last_fullsized":"2024-05-06T14:27:54.590081+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T14:26:41.715500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T14:26:41.715500+0000","last_clean_scrub_stamp":"2024-05-06T14:26:41.715500+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T19:04:40.219350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":19.544,"5min":19.544,"15min":19.544},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.496}]},{"osd":2,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":19.571000000000002,"5min":19.571000000000002,"15min":19.571000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon May 6 14:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:27:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon May 6 14:27:40 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.379}]},{"osd":3,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon May 6 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 6 14:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.437},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33200000000000002},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon May 6 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon May 6 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56699999999999995}]},{"osd":4,"last update":"Mon May 6 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.33400000000000002},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.46700000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.33400000000000002},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon May 6 14:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.309}]},{"osd":3,"last update":"Mon May 6 14:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon May 6 14:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T14:28:44.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-06T14:28:44.985 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:45.356 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:45 smithi153 ceph-mon[27177]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T14:28:45.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:45 smithi190 ceph-mon[33671]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T14:28:46.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:46 smithi153 ceph-mon[27177]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:46.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:46 smithi190 ceph-mon[33671]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 -- 172.21.15.153:0/946298078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae81003d0 msgr2=0x7f7ae81007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 --2- 172.21.15.153:0/946298078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae81003d0 0x7f7ae81007a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000b0a0 tx=0x7f7ae000b3b0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 -- 172.21.15.153:0/946298078 shutdown_connections 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 --2- 172.21.15.153:0/946298078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ae8100d70 0x7f7ae8104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 --2- 172.21.15.153:0/946298078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae81003d0 0x7f7ae81007a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 -- 172.21.15.153:0/946298078 >> 172.21.15.153:0/946298078 conn(0x7f7ae80fbc50 msgr2=0x7f7ae80fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 -- 172.21.15.153:0/946298078 shutdown_connections 2024-05-06T14:28:46.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.751+0000 7f7aefa37700 1 -- 172.21.15.153:0/946298078 wait complete. 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aefa37700 1 Processor -- start 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aefa37700 1 -- start start 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aefa37700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ae81003d0 0x7f7ae8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aefa37700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 0x7f7ae8073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aefa37700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8073ad0 con 0x7f7ae8100d70 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aefa37700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8073c10 con 0x7f7ae81003d0 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aecfd2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 0x7f7ae8073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aecfd2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 0x7f7ae8073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55712/0 (socket says 172.21.15.153:55712) 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aecfd2700 1 -- 172.21.15.153:0/733803914 learned_addr learned my addr 172.21.15.153:0/733803914 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aed7d3700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ae81003d0 0x7f7ae8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aecfd2700 1 -- 172.21.15.153:0/733803914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ae81003d0 msgr2=0x7f7ae8073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aecfd2700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ae81003d0 0x7f7ae8073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.752+0000 7f7aecfd2700 1 -- 172.21.15.153:0/733803914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae0009d00 con 0x7f7ae8100d70 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.753+0000 7f7aecfd2700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 0x7f7ae8073590 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800b480 tx=0x7f7ad800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:46.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.753+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad80118c0 con 0x7f7ae8100d70 2024-05-06T14:28:46.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.753+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ad8010380 con 0x7f7ae8100d70 2024-05-06T14:28:46.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.753+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae81a2030 con 0x7f7ae8100d70 2024-05-06T14:28:46.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.753+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad800f4f0 con 0x7f7ae8100d70 2024-05-06T14:28:46.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.753+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae81a2440 con 0x7f7ae8100d70 2024-05-06T14:28:46.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.754+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae81086e0 con 0x7f7ae8100d70 2024-05-06T14:28:46.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.754+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7ad80104f0 con 0x7f7ae8100d70 2024-05-06T14:28:46.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.755+0000 7f7ade7fc700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7ad406c680 0x7f7ad406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:46.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.755+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7ad808ae70 con 0x7f7ae8100d70 2024-05-06T14:28:46.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.755+0000 7f7aed7d3700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7ad406c680 0x7f7ad406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:46.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.755+0000 7f7aed7d3700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7ad406c680 0x7f7ad406eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000b830 tx=0x7f7ae00097c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:46.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.759+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7ad8055890 con 0x7f7ae8100d70 2024-05-06T14:28:46.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.944+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ae802cd60 con 0x7f7ad406c680 2024-05-06T14:28:46.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.946+0000 7f7ade7fc700 1 -- 172.21.15.153:0/733803914 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28097 (secure 0 0 0) 0x7f7ae802cd60 con 0x7f7ad406c680 2024-05-06T14:28:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7ad406c680 msgr2=0x7f7ad406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7ad406c680 0x7f7ad406eb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000b830 tx=0x7f7ae00097c0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 msgr2=0x7f7ae8073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 0x7f7ae8073590 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800b480 tx=0x7f7ad800b840 comp rx=0 tx=0).stop 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 shutdown_connections 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7ad406c680 0x7f7ad406eb30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ae81003d0 0x7f7ae8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 --2- 172.21.15.153:0/733803914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ae8100d70 0x7f7ae8073590 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:46.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 >> 172.21.15.153:0/733803914 conn(0x7f7ae80fbc50 msgr2=0x7f7ae80fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:46.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 shutdown_connections 2024-05-06T14:28:46.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:46.948+0000 7f7aefa37700 1 -- 172.21.15.153:0/733803914 wait complete. 2024-05-06T14:28:46.936 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-05-06T14:28:47.582 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-05-06T14:28:44.970749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002342"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-05-06T14:27:54.590081+0000","last_change":"2024-05-06T14:27:28.077420+0000","last_active":"2024-05-06T14:27:54.590081+0000","last_peered":"2024-05-06T14:27:54.590081+0000","last_clean":"2024-05-06T14:27:54.590081+0000","last_became_active":"2024-05-06T14:27:28.077132+0000","last_became_peered":"2024-05-06T14:27:28.077132+0000","last_unstale":"2024-05-06T14:27:54.590081+0000","last_undegraded":"2024-05-06T14:27:54.590081+0000","last_fullsized":"2024-05-06T14:27:54.590081+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T14:26:41.715500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T14:26:41.715500+0000","last_clean_scrub_stamp":"2024-05-06T14:26:41.715500+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T19:04:40.219350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":19.544,"5min":19.544,"15min":19.544},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":19.571000000000002,"5min":19.571000000000002,"15min":19.571000000000002},"last":0.503}]},{"osd":3,"last update":"Mon May 6 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon May 6 14:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:27:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon May 6 14:27:40 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.379}]},{"osd":3,"last update":"Mon May 6 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon May 6 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 6 14:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.255},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33200000000000002},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon May 6 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon May 6 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon May 6 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.33400000000000002},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.46700000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.33400000000000002},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon May 6 14:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.309}]},{"osd":3,"last update":"Mon May 6 14:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon May 6 14:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T14:28:47.583 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-06T14:28:47.583 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-06T14:28:47.583 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-06T14:28:47.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-06T14:28:47.788 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:48.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:48 smithi153 ceph-mon[27177]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T14:28:48.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:48 smithi153 ceph-mon[27177]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:48.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:48 smithi190 ceph-mon[33671]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T14:28:48.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:48 smithi190 ceph-mon[33671]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.337+0000 7f133e142700 1 -- 172.21.15.153:0/1972162923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 msgr2=0x7f1338104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.337+0000 7f133e142700 1 --2- 172.21.15.153:0/1972162923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338104d20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f132c009910 tx=0x7f132c009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 -- 172.21.15.153:0/1972162923 shutdown_connections 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 --2- 172.21.15.153:0/1972162923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338104d20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 --2- 172.21.15.153:0/1972162923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13381003b0 0x7f1338100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 -- 172.21.15.153:0/1972162923 >> 172.21.15.153:0/1972162923 conn(0x7f13380fbc30 msgr2=0x7f13380fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 -- 172.21.15.153:0/1972162923 shutdown_connections 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 -- 172.21.15.153:0/1972162923 wait complete. 2024-05-06T14:28:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 Processor -- start 2024-05-06T14:28:49.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.338+0000 7f133e142700 1 -- start start 2024-05-06T14:28:49.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f133e142700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13381003b0 0x7f1338073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:49.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f133e142700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:49.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f133e142700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338073ad0 con 0x7f1338100d50 2024-05-06T14:28:49.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f133e142700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338073c10 con 0x7f13381003b0 2024-05-06T14:28:49.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55736/0 (socket says 172.21.15.153:55736) 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 -- 172.21.15.153:0/1461687698 learned_addr learned my addr 172.21.15.153:0/1461687698 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f13377fe700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13381003b0 0x7f1338073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 -- 172.21.15.153:0/1461687698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13381003b0 msgr2=0x7f1338073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13381003b0 0x7f1338073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 -- 172.21.15.153:0/1461687698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0095f0 con 0x7f1338100d50 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1336ffd700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338073590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f132c0092e0 tx=0x7f132c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132c01d070 con 0x7f1338100d50 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1338073e90 con 0x7f1338100d50 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f132c005dc0 con 0x7f1338100d50 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132c017400 con 0x7f1338100d50 2024-05-06T14:28:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.339+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13381a22c0 con 0x7f1338100d50 2024-05-06T14:28:49.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.340+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338108720 con 0x7f1338100d50 2024-05-06T14:28:49.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.341+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f132c00f460 con 0x7f1338100d50 2024-05-06T14:28:49.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.341+0000 7f1334ff9700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f132006c680 0x7f132006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:49.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.341+0000 7f13377fe700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f132006c680 0x7f132006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:49.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.341+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f132c08c960 con 0x7f1338100d50 2024-05-06T14:28:49.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.342+0000 7f13377fe700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f132006c680 0x7f132006eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1328005950 tx=0x7f132800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:49.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.346+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f132c057380 con 0x7f1338100d50 2024-05-06T14:28:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.578+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f133804ea50 con 0x7f1338100d50 2024-05-06T14:28:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.578+0000 7f1334ff9700 1 -- 172.21.15.153:0/1461687698 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f132c05a9a0 con 0x7f1338100d50 2024-05-06T14:28:49.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:28:49.565 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-06T14:28:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f132006c680 msgr2=0x7f132006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f132006c680 0x7f132006eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1328005950 tx=0x7f132800b410 comp rx=0 tx=0).stop 2024-05-06T14:28:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 msgr2=0x7f1338073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338073590 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f132c0092e0 tx=0x7f132c00bd70 comp rx=0 tx=0).stop 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 shutdown_connections 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f132006c680 0x7f132006eb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13381003b0 0x7f1338073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 --2- 172.21.15.153:0/1461687698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1338100d50 0x7f1338073590 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 >> 172.21.15.153:0/1461687698 conn(0x7f13380fbc30 msgr2=0x7f13380fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.580+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 shutdown_connections 2024-05-06T14:28:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:49.581+0000 7f133e142700 1 -- 172.21.15.153:0/1461687698 wait complete. 2024-05-06T14:28:50.178 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:50 smithi153 ceph-mon[27177]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:50.179 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:50 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1461687698' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T14:28:50.201 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-06T14:28:50.201 INFO:tasks.cephadm:Setup complete, yielding 2024-05-06T14:28:50.202 INFO:teuthology.run_tasks:Running task print... 2024-05-06T14:28:50.211 INFO:teuthology.task.print:**** done end installing v18.2.0 cephadm ... 2024-05-06T14:28:50.211 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:28:50.220 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:28:50.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-05-06T14:28:50.425 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:50.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:50 smithi190 ceph-mon[33671]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:50.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:50 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1461687698' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T14:28:52.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 -- 172.21.15.153:0/3203470810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 msgr2=0x7f2f5c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 --2- 172.21.15.153:0/3203470810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c0ff460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009910 tx=0x7f2f58009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 -- 172.21.15.153:0/3203470810 shutdown_connections 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 --2- 172.21.15.153:0/3203470810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 0x7f2f5c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 --2- 172.21.15.153:0/3203470810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c0ff460 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 -- 172.21.15.153:0/3203470810 >> 172.21.15.153:0/3203470810 conn(0x7f2f5c0fa5e0 msgr2=0x7f2f5c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 -- 172.21.15.153:0/3203470810 shutdown_connections 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.065+0000 7f2f636c9700 1 -- 172.21.15.153:0/3203470810 wait complete. 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f636c9700 1 Processor -- start 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f636c9700 1 -- start start 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f636c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c194910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f636c9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 0x7f2f5c194e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f636c9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c195470 con 0x7f2f5c0ff050 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f636c9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c1955b0 con 0x7f2f5c100250 2024-05-06T14:28:52.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c194910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c194910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40372/0 (socket says 172.21.15.153:40372) 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 -- 172.21.15.153:0/2669744841 learned_addr learned my addr 172.21.15.153:0/2669744841 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f60c64700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 0x7f2f5c194e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 -- 172.21.15.153:0/2669744841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 msgr2=0x7f2f5c194e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 0x7f2f5c194e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 -- 172.21.15.153:0/2669744841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580095f0 con 0x7f2f5c0ff050 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f60c64700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 0x7f2f5c194e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.066+0000 7f2f61465700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c194910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800bda0 tx=0x7f2f5800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.067+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f5801d070 con 0x7f2f5c0ff050 2024-05-06T14:28:52.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.067+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c19a000 con 0x7f2f5c0ff050 2024-05-06T14:28:52.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.067+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2f58004c50 con 0x7f2f5c0ff050 2024-05-06T14:28:52.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.067+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f5800f690 con 0x7f2f5c0ff050 2024-05-06T14:28:52.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.067+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c19a570 con 0x7f2f5c0ff050 2024-05-06T14:28:52.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.068+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5c107c20 con 0x7f2f5c0ff050 2024-05-06T14:28:52.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.068+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f2f580229b0 con 0x7f2f5c0ff050 2024-05-06T14:28:52.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.068+0000 7f2f527fc700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2f4806c680 0x7f2f4806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:52.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.069+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2f5808cc70 con 0x7f2f5c0ff050 2024-05-06T14:28:52.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.069+0000 7f2f60c64700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2f4806c680 0x7f2f4806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:52.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.069+0000 7f2f60c64700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2f4806c680 0x7f2f4806eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0099d0 tx=0x7f2f4c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:52.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.073+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2f5805b1d0 con 0x7f2f5c0ff050 2024-05-06T14:28:52.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.267+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f2f5c107de0 con 0x7f2f5c0ff050 2024-05-06T14:28:52.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.272+0000 7f2f527fc700 1 -- 172.21.15.153:0/2669744841 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f2f5805ad60 con 0x7f2f5c0ff050 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2f4806c680 msgr2=0x7f2f4806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2f4806c680 0x7f2f4806eb30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0099d0 tx=0x7f2f4c008040 comp rx=0 tx=0).stop 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 msgr2=0x7f2f5c194910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c194910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800bda0 tx=0x7f2f5800be80 comp rx=0 tx=0).stop 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 shutdown_connections 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f5c0ff050 0x7f2f5c194910 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f2f4806c680 0x7f2f4806eb30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:52.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 --2- 172.21.15.153:0/2669744841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f5c100250 0x7f2f5c194e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:52.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 >> 172.21.15.153:0/2669744841 conn(0x7f2f5c0fa5e0 msgr2=0x7f2f5c103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:52.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 shutdown_connections 2024-05-06T14:28:52.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:52.274+0000 7f2f636c9700 1 -- 172.21.15.153:0/2669744841 wait complete. 2024-05-06T14:28:52.379 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:52 smithi153 ceph-mon[27177]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:52.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:52 smithi190 ceph-mon[33671]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:52.828 INFO:teuthology.run_tasks:Running task print... 2024-05-06T14:28:52.842 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-05-06T14:28:52.842 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:28:52.862 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:28:52.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch status' 2024-05-06T14:28:53.081 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:53.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:53 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2669744841' entity='client.admin' 2024-05-06T14:28:53.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:28:53.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:28:53.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:28:53.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:53 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:28:53.626 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:53 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2669744841' entity='client.admin' 2024-05-06T14:28:53.626 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:28:53.626 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:28:53.626 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:28:53.626 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:53 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:28:54.467 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:54 smithi153 ceph-mon[27177]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 -- 172.21.15.153:0/3933030319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 msgr2=0x7fa144101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 --2- 172.21.15.153:0/3933030319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144101930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa138009910 tx=0x7fa138009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 -- 172.21.15.153:0/3933030319 shutdown_connections 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 --2- 172.21.15.153:0/3933030319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144101930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 --2- 172.21.15.153:0/3933030319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1440fe460 0x7fa1440fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 -- 172.21.15.153:0/3933030319 >> 172.21.15.153:0/3933030319 conn(0x7fa1440f9db0 msgr2=0x7fa1440fc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 -- 172.21.15.153:0/3933030319 shutdown_connections 2024-05-06T14:28:54.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.681+0000 7fa1490c4700 1 -- 172.21.15.153:0/3933030319 wait complete. 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.682+0000 7fa1490c4700 1 Processor -- start 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa1490c4700 1 -- start start 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa1490c4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1440fe460 0x7fa144195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa1490c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa1490c4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1441964a0 con 0x7fa1441014e0 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa1490c4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1441965e0 con 0x7fa1440fe460 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40392/0 (socket says 172.21.15.153:40392) 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 -- 172.21.15.153:0/2938633872 learned_addr learned my addr 172.21.15.153:0/2938633872 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa142d9d700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1440fe460 0x7fa144195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:54.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 -- 172.21.15.153:0/2938633872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1440fe460 msgr2=0x7fa144195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1440fe460 0x7fa144195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 -- 172.21.15.153:0/2938633872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1380095f0 con 0x7fa1441014e0 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.683+0000 7fa14259c700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144195e80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa13800bd70 tx=0x7fa13800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.684+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa13801d070 con 0x7fa1441014e0 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.684+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14419b030 con 0x7fa1441014e0 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.684+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa138022470 con 0x7fa1441014e0 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.684+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa13800f670 con 0x7fa1441014e0 2024-05-06T14:28:54.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.684+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14419b520 con 0x7fa1441014e0 2024-05-06T14:28:54.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.685+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa144108cc0 con 0x7fa1441014e0 2024-05-06T14:28:54.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.685+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa138022a50 con 0x7fa1441014e0 2024-05-06T14:28:54.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.685+0000 7fa12bfff700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa12c06c680 0x7fa12c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:54.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.686+0000 7fa142d9d700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa12c06c680 0x7fa12c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:54.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.686+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa13808cc10 con 0x7fa1441014e0 2024-05-06T14:28:54.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.686+0000 7fa142d9d700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa12c06c680 0x7fa12c06eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa134005fd0 tx=0x7fa134005dd0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:54.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.690+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa13805b170 con 0x7fa1441014e0 2024-05-06T14:28:54.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:54 smithi190 ceph-mon[33671]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.893+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa144105e30 con 0x7fa12c06c680 2024-05-06T14:28:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.894+0000 7fa12bfff700 1 -- 172.21.15.153:0/2938633872 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa144105e30 con 0x7fa12c06c680 2024-05-06T14:28:54.883 INFO:teuthology.orchestra.run.smithi153.stdout:Backend: cephadm 2024-05-06T14:28:54.883 INFO:teuthology.orchestra.run.smithi153.stdout:Available: Yes 2024-05-06T14:28:54.883 INFO:teuthology.orchestra.run.smithi153.stdout:Paused: No 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa12c06c680 msgr2=0x7fa12c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa12c06c680 0x7fa12c06eb30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa134005fd0 tx=0x7fa134005dd0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 msgr2=0x7fa144195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144195e80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa13800bd70 tx=0x7fa13800be50 comp rx=0 tx=0).stop 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 shutdown_connections 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa12c06c680 0x7fa12c06eb30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1440fe460 0x7fa144195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 --2- 172.21.15.153:0/2938633872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441014e0 0x7fa144195e80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 >> 172.21.15.153:0/2938633872 conn(0x7fa1440f9db0 msgr2=0x7fa144104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:54.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 shutdown_connections 2024-05-06T14:28:54.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:54.896+0000 7fa1490c4700 1 -- 172.21.15.153:0/2938633872 wait complete. 2024-05-06T14:28:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:55.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-06T14:28:55.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:28:55.776 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:56.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:56 smithi153 ceph-mon[27177]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:28:56.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:56 smithi153 ceph-mon[27177]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:56.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:56 smithi190 ceph-mon[33671]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:28:56.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:56 smithi190 ceph-mon[33671]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 -- 172.21.15.153:0/2961448466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 msgr2=0x7f82841017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 --2- 172.21.15.153:0/2961448466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f82841017e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8274009910 tx=0x7f8274009c20 comp rx=0 tx=0).stop 2024-05-06T14:28:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 -- 172.21.15.153:0/2961448466 shutdown_connections 2024-05-06T14:28:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 --2- 172.21.15.153:0/2961448466 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f82841017e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:57.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 --2- 172.21.15.153:0/2961448466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 0x7f82841005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 -- 172.21.15.153:0/2961448466 >> 172.21.15.153:0/2961448466 conn(0x7f82840fb740 msgr2=0x7f82840fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 -- 172.21.15.153:0/2961448466 shutdown_connections 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.402+0000 7f8288c34700 1 -- 172.21.15.153:0/2961448466 wait complete. 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8288c34700 1 Processor -- start 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8288c34700 1 -- start start 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8288c34700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 0x7f8284195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8288c34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f8284195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8288c34700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82841964c0 con 0x7f8284101390 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8288c34700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284196600 con 0x7f8284100190 2024-05-06T14:28:57.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8281d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f8284195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8281d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f8284195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40418/0 (socket says 172.21.15.153:40418) 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f8281d9b700 1 -- 172.21.15.153:0/1631794843 learned_addr learned my addr 172.21.15.153:0/1631794843 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.403+0000 7f828259c700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 0x7f8284195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f8281d9b700 1 -- 172.21.15.153:0/1631794843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 msgr2=0x7f8284195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f8281d9b700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 0x7f8284195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f8281d9b700 1 -- 172.21.15.153:0/1631794843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82740095f0 con 0x7f8284101390 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f828259c700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 0x7f8284195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f8281d9b700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f8284195ea0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8274005b40 tx=0x7f827400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:57.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f827401d070 con 0x7f8284101390 2024-05-06T14:28:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8274022470 con 0x7f8284101390 2024-05-06T14:28:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828419b050 con 0x7f8284101390 2024-05-06T14:28:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f827400f690 con 0x7f8284101390 2024-05-06T14:28:57.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.404+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828419b540 con 0x7f8284101390 2024-05-06T14:28:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.405+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8284108ce0 con 0x7f8284101390 2024-05-06T14:28:57.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.405+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f82740225e0 con 0x7f8284101390 2024-05-06T14:28:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.406+0000 7f827b7fe700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f827006c680 0x7f827006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:28:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.406+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f827408cd80 con 0x7f8284101390 2024-05-06T14:28:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.406+0000 7f828259c700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f827006c680 0x7f827006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:28:57.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.406+0000 7f828259c700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f827006c680 0x7f827006eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f82841011f0 tx=0x7f826c005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:28:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.410+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f827405b2b0 con 0x7f8284101390 2024-05-06T14:28:57.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.610+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8284105ce0 con 0x7f827006c680 2024-05-06T14:28:57.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.619+0000 7f827b7fe700 1 -- 172.21.15.153:0/1631794843 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f8284105ce0 con 0x7f827006c680 2024-05-06T14:28:57.606 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (3m) 2m ago 5m 26.3M - 0.25.0 c8568f914cd2 5869e5f8b16f 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi153 smithi153 running (5m) 2m ago 5m 7025k - 18.2.0 dc2bc1663786 9903f06d2ec9 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi190 smithi190 running (4m) 64s ago 4m 6966k - 18.2.0 dc2bc1663786 8fa6eb9bb609 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (5m) 2m ago 5m 6660k - 18.2.0 dc2bc1663786 d9ddce025318 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi190 smithi190 running (4m) 64s ago 4m 6669k - 18.2.0 dc2bc1663786 1b44eaee674c 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (3m) 2m ago 4m 76.3M - 9.4.7 954c08fa6188 d15dfe6401b3 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.dolfed smithi153 *:9283,8765,8443 running (6m) 2m ago 6m 476M - 18.2.0 dc2bc1663786 0562ae98d3da 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi190.qauohf smithi190 *:8443,9283,8765 running (4m) 64s ago 4m 436M - 18.2.0 dc2bc1663786 1cd612378142 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (6m) 2m ago 6m 44.0M 2048M 18.2.0 dc2bc1663786 44f2993c90f4 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi190 smithi190 running (4m) 64s ago 4m 40.4M 2048M 18.2.0 dc2bc1663786 88aeab4dfa5e 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 2m ago 5m 15.1M - 1.5.0 0da6a335fe13 5c569e0ae0c9 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi190 smithi190 *:9100 running (4m) 64s ago 4m 17.0M - 1.5.0 0da6a335fe13 d1be4aed7b03 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 2m ago 3m 37.6M 3946M 18.2.0 dc2bc1663786 6c202dfae018 2024-05-06T14:28:57.607 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 2m ago 2m 39.3M 3946M 18.2.0 dc2bc1663786 78ff3749996c 2024-05-06T14:28:57.608 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 2m ago 2m 40.6M 3946M 18.2.0 dc2bc1663786 3fbd486b6ac6 2024-05-06T14:28:57.608 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi190 running (2m) 64s ago 2m 42.8M 4970M 18.2.0 dc2bc1663786 79af272339eb 2024-05-06T14:28:57.608 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi190 running (96s) 64s ago 95s 39.3M 4970M 18.2.0 dc2bc1663786 6db95b0197f1 2024-05-06T14:28:57.608 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi190 running (70s) 64s ago 69s 11.5M 4970M 18.2.0 dc2bc1663786 ae3651b2db8d 2024-05-06T14:28:57.608 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (3m) 2m ago 4m 41.7M - 2.43.0 a07b618ecd1d c8f2e7c500cf 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f827006c680 msgr2=0x7f827006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f827006c680 0x7f827006eb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f82841011f0 tx=0x7f826c005f90 comp rx=0 tx=0).stop 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 msgr2=0x7f8284195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f8284195ea0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8274005b40 tx=0x7f827400be20 comp rx=0 tx=0).stop 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 shutdown_connections 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f827006c680 0x7f827006eb30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8284100190 0x7f8284195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 --2- 172.21.15.153:0/1631794843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8284101390 0x7f8284195ea0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 >> 172.21.15.153:0/1631794843 conn(0x7f82840fb740 msgr2=0x7f82841045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 shutdown_connections 2024-05-06T14:28:57.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:28:57.621+0000 7f8288c34700 1 -- 172.21.15.153:0/1631794843 wait complete. 2024-05-06T14:28:58.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-06T14:28:58.460 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:28:58.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:58 smithi153 ceph-mon[27177]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:58.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:58 smithi190 ceph-mon[33671]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:28:59.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:28:59 smithi153 ceph-mon[27177]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:28:59.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:28:59 smithi190 ceph-mon[33671]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:00.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2114650024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048103520 msgr2=0x7fa048105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2114650024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048103520 0x7fa048105900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa038009910 tx=0x7fa038009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2114650024 shutdown_connections 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2114650024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048103520 0x7fa048105900 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2114650024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048100c00 0x7fa048102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2114650024 >> 172.21.15.153:0/2114650024 conn(0x7fa0480fa600 msgr2=0x7fa0480fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2114650024 shutdown_connections 2024-05-06T14:29:00.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.107+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2114650024 wait complete. 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04fcdb700 1 Processor -- start 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04fcdb700 1 -- start start 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04fcdb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 0x7fa0481959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04fcdb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048103520 0x7fa048195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04fcdb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa048196530 con 0x7fa048100c00 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04fcdb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa048196670 con 0x7fa048103520 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04da77700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 0x7fa0481959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04da77700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 0x7fa0481959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33628/0 (socket says 172.21.15.153:33628) 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04da77700 1 -- 172.21.15.153:0/2535368200 learned_addr learned my addr 172.21.15.153:0/2535368200 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.108+0000 7fa04d276700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048103520 0x7fa048195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04da77700 1 -- 172.21.15.153:0/2535368200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048103520 msgr2=0x7fa048195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04da77700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048103520 0x7fa048195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:00.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04da77700 1 -- 172.21.15.153:0/2535368200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0380095f0 con 0x7fa048100c00 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04d276700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048103520 0x7fa048195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04da77700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 0x7fa0481959d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa04400b4b0 tx=0x7fa04400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa044011800 con 0x7fa048100c00 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa044011e10 con 0x7fa048100c00 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04819b120 con 0x7fa048100c00 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa04400f2e0 con 0x7fa048100c00 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.109+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04819b6f0 con 0x7fa048100c00 2024-05-06T14:29:00.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.110+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04818fc10 con 0x7fa048100c00 2024-05-06T14:29:00.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.110+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa044010380 con 0x7fa048100c00 2024-05-06T14:29:00.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.111+0000 7fa03effd700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa03406c680 0x7fa03406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:00.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.111+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa04408b1a0 con 0x7fa048100c00 2024-05-06T14:29:00.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.111+0000 7fa04d276700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa03406c680 0x7fa03406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:00.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.111+0000 7fa04d276700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa03406c680 0x7fa03406eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa0380059d0 tx=0x7fa038005920 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:00.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.115+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa044059650 con 0x7fa048100c00 2024-05-06T14:29:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.314+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa048061190 con 0x7fa03406c680 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.319+0000 7fa03effd700 1 -- 172.21.15.153:0/2535368200 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7fa048061190 con 0x7fa03406c680 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter 2/2 2m ago 6m * 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:crash 2/2 2m ago 6m * 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-05-06T14:29:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:mgr 2/2 2m ago 6m count:2 2024-05-06T14:29:00.307 INFO:teuthology.orchestra.run.smithi153.stdout:mon 2/2 2m ago 5m smithi153:172.21.15.153=smithi153;smithi190:172.21.15.190=smithi190;count:2 2024-05-06T14:29:00.307 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-05-06T14:29:00.307 INFO:teuthology.orchestra.run.smithi153.stdout:osd 6 2m ago - 2024-05-06T14:29:00.307 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-05-06T14:29:00.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa03406c680 msgr2=0x7fa03406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:00.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa03406c680 0x7fa03406eb30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa0380059d0 tx=0x7fa038005920 comp rx=0 tx=0).stop 2024-05-06T14:29:00.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 msgr2=0x7fa0481959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:00.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 0x7fa0481959d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa04400b4b0 tx=0x7fa04400b870 comp rx=0 tx=0).stop 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 shutdown_connections 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048100c00 0x7fa0481959d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa03406c680 0x7fa03406eb30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 --2- 172.21.15.153:0/2535368200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa048103520 0x7fa048195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 >> 172.21.15.153:0/2535368200 conn(0x7fa0480fa600 msgr2=0x7fa0480fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 shutdown_connections 2024-05-06T14:29:00.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:00.321+0000 7fa04fcdb700 1 -- 172.21.15.153:0/2535368200 wait complete. 2024-05-06T14:29:00.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:00 smithi190 ceph-mon[33671]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:00.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:00 smithi153 ceph-mon[27177]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:00.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-06T14:29:01.182 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:01.550 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:01 smithi153 ceph-mon[27177]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:01.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:01 smithi190 ceph-mon[33671]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:02.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:02 smithi153 ceph-mon[27177]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:02.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:02 smithi190 ceph-mon[33671]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:02.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.959+0000 7f8b7375e700 1 -- 172.21.15.153:0/3455936544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c100280 msgr2=0x7f8b6c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:02.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.959+0000 7f8b7375e700 1 --2- 172.21.15.153:0/3455936544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c100280 0x7f8b6c1006d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009910 tx=0x7f8b5c009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:02.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- 172.21.15.153:0/3455936544 shutdown_connections 2024-05-06T14:29:02.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 --2- 172.21.15.153:0/3455936544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c100280 0x7f8b6c1006d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:02.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 --2- 172.21.15.153:0/3455936544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c0ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:02.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- 172.21.15.153:0/3455936544 >> 172.21.15.153:0/3455936544 conn(0x7f8b6c0fa630 msgr2=0x7f8b6c0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- 172.21.15.153:0/3455936544 shutdown_connections 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- 172.21.15.153:0/3455936544 wait complete. 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 Processor -- start 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- start start 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c100280 0x7f8b6c101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c101ba0 con 0x7f8b6c0ff080 2024-05-06T14:29:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.960+0000 7f8b7375e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c101ce0 con 0x7f8b6c100280 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c103010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33636/0 (socket says 172.21.15.153:33636) 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 -- 172.21.15.153:0/1128443328 learned_addr learned my addr 172.21.15.153:0/1128443328 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b70cf9700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c100280 0x7f8b6c101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 -- 172.21.15.153:0/1128443328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c100280 msgr2=0x7f8b6c101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c100280 0x7f8b6c101660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 -- 172.21.15.153:0/1128443328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c0095f0 con 0x7f8b6c0ff080 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b714fa700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c103010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800c8f0 tx=0x7f8b6800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:02.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b680078e0 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b6800f450 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c101fc0 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b68018610 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.961+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c19da20 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.962+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c107bd0 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.963+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f8b6800f5c0 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.963+0000 7f8b627fc700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f8b5806c680 0x7f8b5806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.963+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8b6808b330 con 0x7f8b6c0ff080 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.963+0000 7f8b70cf9700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f8b5806c680 0x7f8b5806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:02.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.963+0000 7f8b70cf9700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f8b5806c680 0x7f8b5806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c006010 tx=0x7f8b5c00b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:02.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:02.967+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8b68059860 con 0x7f8b6c0ff080 2024-05-06T14:29:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.164+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b6c0694a0 con 0x7f8b5806c680 2024-05-06T14:29:03.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.166+0000 7f8b627fc700 1 -- 172.21.15.153:0/1128443328 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f8b6c0694a0 con 0x7f8b5806c680 2024-05-06T14:29:03.153 INFO:teuthology.orchestra.run.smithi153.stdout:HOST ADDR LABELS STATUS 2024-05-06T14:29:03.153 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153 172.21.15.153 2024-05-06T14:29:03.153 INFO:teuthology.orchestra.run.smithi153.stdout:smithi190 172.21.15.190 2024-05-06T14:29:03.153 INFO:teuthology.orchestra.run.smithi153.stdout:2 hosts in cluster 2024-05-06T14:29:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f8b5806c680 msgr2=0x7f8b5806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f8b5806c680 0x7f8b5806eb30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c006010 tx=0x7f8b5c00b540 comp rx=0 tx=0).stop 2024-05-06T14:29:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 msgr2=0x7f8b6c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c103010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800c8f0 tx=0x7f8b6800cc00 comp rx=0 tx=0).stop 2024-05-06T14:29:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 shutdown_connections 2024-05-06T14:29:03.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6c0ff080 0x7f8b6c103010 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f8b5806c680 0x7f8b5806eb30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 --2- 172.21.15.153:0/1128443328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c100280 0x7f8b6c101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 >> 172.21.15.153:0/1128443328 conn(0x7f8b6c0fa630 msgr2=0x7f8b6c068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 shutdown_connections 2024-05-06T14:29:03.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:03.168+0000 7f8b7375e700 1 -- 172.21.15.153:0/1128443328 wait complete. 2024-05-06T14:29:03.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-06T14:29:04.079 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:04.496 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:04 smithi153 ceph-mon[27177]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:04.496 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:04 smithi153 ceph-mon[27177]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:04.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:04 smithi190 ceph-mon[33671]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:04.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:04 smithi190 ceph-mon[33671]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:05.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.702+0000 7fa7ac496700 1 -- 172.21.15.153:0/2399638755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 msgr2=0x7fa7a4103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:05.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.702+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2399638755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4103060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009910 tx=0x7fa7a0009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:05.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.702+0000 7fa7ac496700 1 -- 172.21.15.153:0/2399638755 shutdown_connections 2024-05-06T14:29:05.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.702+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2399638755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a41035a0 0x7fa7a4105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.702+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2399638755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4103060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.702+0000 7fa7ac496700 1 -- 172.21.15.153:0/2399638755 >> 172.21.15.153:0/2399638755 conn(0x7fa7a40fa620 msgr2=0x7fa7a40fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 -- 172.21.15.153:0/2399638755 shutdown_connections 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 -- 172.21.15.153:0/2399638755 wait complete. 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 Processor -- start 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 -- start start 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a41035a0 0x7fa7a4195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a41964d0 con 0x7fa7a4100c80 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7ac496700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4196610 con 0x7fa7a41035a0 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7aa232700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7aa232700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33642/0 (socket says 172.21.15.153:33642) 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.703+0000 7fa7aa232700 1 -- 172.21.15.153:0/2292245905 learned_addr learned my addr 172.21.15.153:0/2292245905 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:05.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7a9a31700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a41035a0 0x7fa7a4195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7aa232700 1 -- 172.21.15.153:0/2292245905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a41035a0 msgr2=0x7fa7a4195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7aa232700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a41035a0 0x7fa7a4195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7aa232700 1 -- 172.21.15.153:0/2292245905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a00095f0 con 0x7fa7a4100c80 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7aa232700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4195970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0006010 tx=0x7fa7a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7a001d070 con 0x7fa7a4100c80 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa7a000f460 con 0x7fa7a4100c80 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a419b060 con 0x7fa7a4100c80 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7a0021640 con 0x7fa7a4100c80 2024-05-06T14:29:05.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.704+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a419b550 con 0x7fa7a4100c80 2024-05-06T14:29:05.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.705+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a418fbb0 con 0x7fa7a4100c80 2024-05-06T14:29:05.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.706+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa7a000fab0 con 0x7fa7a4100c80 2024-05-06T14:29:05.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.706+0000 7fa79b7fe700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa79006c680 0x7fa79006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:05.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.706+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa7a008c9a0 con 0x7fa7a4100c80 2024-05-06T14:29:05.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.706+0000 7fa7a9a31700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa79006c680 0x7fa79006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:05.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.707+0000 7fa7a9a31700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa79006c680 0x7fa79006eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa794005fd0 tx=0x7fa79400a560 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:05.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.711+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa7a005aed0 con 0x7fa7a4100c80 2024-05-06T14:29:05.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.906+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7a4061190 con 0x7fa79006c680 2024-05-06T14:29:05.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.908+0000 7fa79b7fe700 1 -- 172.21.15.153:0/2292245905 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fa7a4061190 con 0x7fa79006c680 2024-05-06T14:29:05.895 INFO:teuthology.orchestra.run.smithi153.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-06T14:29:05.895 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 2m ago LVM detected, locked 2024-05-06T14:29:05.895 INFO:teuthology.orchestra.run.smithi153.stdout:smithi190 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT63210012400BGN 372G No 69s ago LVM detected, locked 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa79006c680 msgr2=0x7fa79006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa79006c680 0x7fa79006eb30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa794005fd0 tx=0x7fa79400a560 comp rx=0 tx=0).stop 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 msgr2=0x7fa7a4195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4195970 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0006010 tx=0x7fa7a000be20 comp rx=0 tx=0).stop 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 shutdown_connections 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7a4100c80 0x7fa7a4195970 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa79006c680 0x7fa79006eb30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 --2- 172.21.15.153:0/2292245905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a41035a0 0x7fa7a4195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 >> 172.21.15.153:0/2292245905 conn(0x7fa7a40fa620 msgr2=0x7fa7a40fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 shutdown_connections 2024-05-06T14:29:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:05.910+0000 7fa7ac496700 1 -- 172.21.15.153:0/2292245905 wait complete. 2024-05-06T14:29:06.479 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:29:06.490 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:29:06.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-05-06T14:29:06.513 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:06 smithi153 ceph-mon[27177]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:06.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:06 smithi190 ceph-mon[33671]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:06.696 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:07.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:07 smithi153 ceph-mon[27177]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:07.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:07 smithi190 ceph-mon[33671]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:08.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:08 smithi153 ceph-mon[27177]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 -- 172.21.15.153:0/1241880012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98541013f0 msgr2=0x7f9854101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 --2- 172.21.15.153:0/1241880012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98541013f0 0x7f9854101840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9844009910 tx=0x7f9844009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 -- 172.21.15.153:0/1241880012 shutdown_connections 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 --2- 172.21.15.153:0/1241880012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98541013f0 0x7f9854101840 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 --2- 172.21.15.153:0/1241880012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98540fe410 0x7f98540fe820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 -- 172.21.15.153:0/1241880012 >> 172.21.15.153:0/1241880012 conn(0x7f98540f9d80 msgr2=0x7f98540fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 -- 172.21.15.153:0/1241880012 shutdown_connections 2024-05-06T14:29:08.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.329+0000 7f98597bc700 1 -- 172.21.15.153:0/1241880012 wait complete. 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98597bc700 1 Processor -- start 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98597bc700 1 -- start start 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98597bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 0x7f9854100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98597bc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98541013f0 0x7f98540ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98597bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540ff8e0 con 0x7f98540fe410 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98597bc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540ffa20 con 0x7f98541013f0 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f9852ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 0x7f9854100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f9852ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 0x7f9854100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33656/0 (socket says 172.21.15.153:33656) 2024-05-06T14:29:08.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f9852ffd700 1 -- 172.21.15.153:0/704408491 learned_addr learned my addr 172.21.15.153:0/704408491 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.330+0000 7f98527fc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98541013f0 0x7f98540ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f9852ffd700 1 -- 172.21.15.153:0/704408491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98541013f0 msgr2=0x7f98540ff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f9852ffd700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98541013f0 0x7f98540ff3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f9852ffd700 1 -- 172.21.15.153:0/704408491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98440095f0 con 0x7f98540fe410 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f9852ffd700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 0x7f9854100d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f984800b480 tx=0x7f984800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9848011800 con 0x7f98540fe410 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98540ffd00 con 0x7f98540fe410 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9848011e10 con 0x7f98540fe410 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9848010ea0 con 0x7f98540fe410 2024-05-06T14:29:08.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.331+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985419da60 con 0x7f98540fe410 2024-05-06T14:29:08.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.332+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9854108d30 con 0x7f98540fe410 2024-05-06T14:29:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.333+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f9848011960 con 0x7f98540fe410 2024-05-06T14:29:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.333+0000 7f983bfff700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f983c06c680 0x7f983c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.333+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f984808b0e0 con 0x7f98540fe410 2024-05-06T14:29:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.333+0000 7f98527fc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f983c06c680 0x7f983c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.333+0000 7f98527fc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f983c06c680 0x7f983c06eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9844006010 tx=0x7f984400b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:08.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.337+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9848059590 con 0x7f98540fe410 2024-05-06T14:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:08.560+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9854105d40 con 0x7f983c06c680 2024-05-06T14:29:08.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:08 smithi190 ceph-mon[33671]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:09.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:09 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-05-06T14:29:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:09 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-05-06T14:29:10.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.347+0000 7f983bfff700 1 -- 172.21.15.153:0/704408491 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9854105d40 con 0x7f983c06c680 2024-05-06T14:29:10.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f983c06c680 msgr2=0x7f983c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:10.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f983c06c680 0x7f983c06eb30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9844006010 tx=0x7f984400b540 comp rx=0 tx=0).stop 2024-05-06T14:29:10.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 msgr2=0x7f9854100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 0x7f9854100d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f984800b480 tx=0x7f984800b790 comp rx=0 tx=0).stop 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 shutdown_connections 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98540fe410 0x7f9854100d50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f983c06c680 0x7f983c06eb30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 --2- 172.21.15.153:0/704408491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98541013f0 0x7f98540ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.349+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 >> 172.21.15.153:0/704408491 conn(0x7f98540f9d80 msgr2=0x7f9854104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.350+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 shutdown_connections 2024-05-06T14:29:10.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:10.350+0000 7f98597bc700 1 -- 172.21.15.153:0/704408491 wait complete. 2024-05-06T14:29:10.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:10 smithi190 ceph-mon[33671]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:10 smithi190 ceph-mon[33671]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-05-06T14:29:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:10 smithi190 ceph-mon[33671]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T14:29:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-05-06T14:29:10.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:10.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-mon[27177]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:10.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-mon[27177]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:10.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-05-06T14:29:10.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-mon[27177]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T14:29:10.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-05-06T14:29:10.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:10.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:10 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153[27173]: 2024-05-06T14:29:10.325+0000 7f9b56660700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-06T14:29:11.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-06T14:29:11.566 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-05-06T14:29:11.566 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: osdmap e43: 6 total, 6 up, 6 in 2024-05-06T14:29:11.566 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-05-06T14:29:11.566 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-06T14:29:11.566 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: osdmap e44: 6 total, 6 up, 6 in 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: fsmap cephfs:0 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: Saving service mds.cephfs spec with placement count:4 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.hiwroq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:11.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.hiwroq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:11.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: osdmap e43: 6 total, 6 up, 6 in 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: osdmap e44: 6 total, 6 up, 6 in 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: fsmap cephfs:0 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: Saving service mds.cephfs spec with placement count:4 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:11.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:29:11.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:11.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:29:11.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:11.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.hiwroq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:11.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.hiwroq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:11.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:11.809 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:12.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:12 smithi153 ceph-mon[27177]: Deploying daemon mds.cephfs.smithi153.hiwroq on smithi153 2024-05-06T14:29:12.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:12 smithi153 ceph-mon[27177]: pgmap v175: 65 pgs: 16 creating+peering, 6 active+clean, 43 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:12.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:12 smithi153 ceph-mon[27177]: osdmap e45: 6 total, 6 up, 6 in 2024-05-06T14:29:12.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:12 smithi190 ceph-mon[33671]: Deploying daemon mds.cephfs.smithi153.hiwroq on smithi153 2024-05-06T14:29:12.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:12 smithi190 ceph-mon[33671]: pgmap v175: 65 pgs: 16 creating+peering, 6 active+clean, 43 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:12.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:12 smithi190 ceph-mon[33671]: osdmap e45: 6 total, 6 up, 6 in 2024-05-06T14:29:13.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:13 smithi153 ceph-mon[27177]: osdmap e46: 6 total, 6 up, 6 in 2024-05-06T14:29:13.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:13 smithi190 ceph-mon[33671]: osdmap e46: 6 total, 6 up, 6 in 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 -- 172.21.15.153:0/2727585286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248101370 msgr2=0x7f02481017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 --2- 172.21.15.153:0/2727585286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248101370 0x7f02481017c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f023c009910 tx=0x7f023c009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 -- 172.21.15.153:0/2727585286 shutdown_connections 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 --2- 172.21.15.153:0/2727585286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248101370 0x7f02481017c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 --2- 172.21.15.153:0/2727585286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248100170 0x7f0248100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 -- 172.21.15.153:0/2727585286 >> 172.21.15.153:0/2727585286 conn(0x7f02480fb6e0 msgr2=0x7f02480fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 -- 172.21.15.153:0/2727585286 shutdown_connections 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 -- 172.21.15.153:0/2727585286 wait complete. 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 Processor -- start 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 -- start start 2024-05-06T14:29:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.087+0000 7f024ea50700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 0x7f0248195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f024ea50700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248101370 0x7f0248195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f024ea50700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02481964e0 con 0x7f0248100170 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f024ea50700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248196620 con 0x7f0248101370 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 0x7f0248195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 0x7f0248195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52052/0 (socket says 172.21.15.153:52052) 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 -- 172.21.15.153:0/3383673786 learned_addr learned my addr 172.21.15.153:0/3383673786 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f02477fe700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248101370 0x7f0248195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 -- 172.21.15.153:0/3383673786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248101370 msgr2=0x7f0248195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248101370 0x7f0248195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 -- 172.21.15.153:0/3383673786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c0095f0 con 0x7f0248100170 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f0247fff700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 0x7f0248195980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f023800c8f0 tx=0x7f023800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0238007a70 con 0x7f0248100170 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024819b0d0 con 0x7f0248100170 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0238007bd0 con 0x7f0248100170 2024-05-06T14:29:14.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0238018630 con 0x7f0248100170 2024-05-06T14:29:14.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.088+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024819b620 con 0x7f0248100170 2024-05-06T14:29:14.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.089+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0248108cc0 con 0x7f0248100170 2024-05-06T14:29:14.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.090+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f023800f450 con 0x7f0248100170 2024-05-06T14:29:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.090+0000 7f02457fa700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f023006c680 0x7f023006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.090+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f023808b850 con 0x7f0248100170 2024-05-06T14:29:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.090+0000 7f02477fe700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f023006c680 0x7f023006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:14.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.091+0000 7f02477fe700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f023006c680 0x7f023006eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f023c00b5c0 tx=0x7f023c005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:14.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.095+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0238059a70 con 0x7f0248100170 2024-05-06T14:29:14.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.336+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f024819c0a0 con 0x7f0248100170 2024-05-06T14:29:14.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.337+0000 7f02457fa700 1 -- 172.21.15.153:0/3383673786 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7f0238059600 con 0x7f0248100170 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:e2 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 2 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-05-06T14:29:14.324 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-05-06T14:29:10.325593+0000 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-05-06T14:29:10.325648+0000 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 0 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 1 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:in 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-05-06T14:29:14.325 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 0 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f023006c680 msgr2=0x7f023006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f023006c680 0x7f023006eb30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f023c00b5c0 tx=0x7f023c005960 comp rx=0 tx=0).stop 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 msgr2=0x7f0248195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 0x7f0248195980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f023800c8f0 tx=0x7f023800ccb0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 shutdown_connections 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0248100170 0x7f0248195980 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f023006c680 0x7f023006eb30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 --2- 172.21.15.153:0/3383673786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0248101370 0x7f0248195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:14.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 >> 172.21.15.153:0/3383673786 conn(0x7f02480fb6e0 msgr2=0x7f02481045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:14.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 shutdown_connections 2024-05-06T14:29:14.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:14.339+0000 7f024ea50700 1 -- 172.21.15.153:0/3383673786 wait complete. 2024-05-06T14:29:14.328 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 2 2024-05-06T14:29:14.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:14 smithi190 ceph-mon[33671]: pgmap v178: 65 pgs: 16 creating+peering, 33 active+clean, 16 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:14.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:14 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3383673786' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-06T14:29:14.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:14 smithi153 ceph-mon[27177]: pgmap v178: 65 pgs: 16 creating+peering, 33 active+clean, 16 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:14.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:14 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3383673786' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-06T14:29:15.157 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:29:15.171 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:29:15.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-05-06T14:29:15.385 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.psnzbz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.psnzbz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: Deploying daemon mds.cephfs.smithi190.psnzbz on smithi190 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: pgmap v179: 65 pgs: 13 creating+peering, 48 active+clean, 4 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:15.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6826/4248251632,v1:172.21.15.153:6827/4248251632] up:boot 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: daemon mds.cephfs.smithi153.hiwroq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: Cluster is now healthy 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: fsmap cephfs:0 1 up:standby 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.hiwroq"}]: dispatch 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: fsmap cephfs:1 {0=cephfs.smithi153.hiwroq=up:creating} 2024-05-06T14:29:15.687 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:15 smithi190 ceph-mon[33671]: daemon mds.cephfs.smithi153.hiwroq is now active in filesystem cephfs as rank 0 2024-05-06T14:29:15.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:15.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:15.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:15.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.psnzbz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:15.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.psnzbz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:15.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: Deploying daemon mds.cephfs.smithi190.psnzbz on smithi190 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: pgmap v179: 65 pgs: 13 creating+peering, 48 active+clean, 4 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6826/4248251632,v1:172.21.15.153:6827/4248251632] up:boot 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: daemon mds.cephfs.smithi153.hiwroq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: Cluster is now healthy 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: fsmap cephfs:0 1 up:standby 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.hiwroq"}]: dispatch 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: fsmap cephfs:1 {0=cephfs.smithi153.hiwroq=up:creating} 2024-05-06T14:29:15.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:15 smithi153 ceph-mon[27177]: daemon mds.cephfs.smithi153.hiwroq is now active in filesystem cephfs as rank 0 2024-05-06T14:29:16.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 -- 172.21.15.153:0/367530822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f8100270 msgr2=0x7ff3f81006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:16.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 --2- 172.21.15.153:0/367530822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f8100270 0x7ff3f81006c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009910 tx=0x7ff3e8009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:16.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 -- 172.21.15.153:0/367530822 shutdown_connections 2024-05-06T14:29:16.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 --2- 172.21.15.153:0/367530822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f8100270 0x7ff3f81006c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:16.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 --2- 172.21.15.153:0/367530822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f80ff070 0x7ff3f80ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:16.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 -- 172.21.15.153:0/367530822 >> 172.21.15.153:0/367530822 conn(0x7ff3f80fa600 msgr2=0x7ff3f80fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 -- 172.21.15.153:0/367530822 shutdown_connections 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.878+0000 7ff3fc805700 1 -- 172.21.15.153:0/367530822 wait complete. 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.879+0000 7ff3fc805700 1 Processor -- start 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.879+0000 7ff3fc805700 1 -- start start 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.879+0000 7ff3fc805700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 0x7ff3f8103000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.879+0000 7ff3fc805700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100270 0x7ff3f8101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.879+0000 7ff3fc805700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f8101b90 con 0x7ff3f80ff070 2024-05-06T14:29:16.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.879+0000 7ff3fc805700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f8101cd0 con 0x7ff3f8100270 2024-05-06T14:29:16.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 0x7ff3f8103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:16.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 0x7ff3f8103000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52104/0 (socket says 172.21.15.153:52104) 2024-05-06T14:29:16.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 -- 172.21.15.153:0/360769680 learned_addr learned my addr 172.21.15.153:0/360769680 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:16.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f5d9b700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100270 0x7ff3f8101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 -- 172.21.15.153:0/360769680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100270 msgr2=0x7ff3f8101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100270 0x7ff3f8101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 -- 172.21.15.153:0/360769680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e80095f0 con 0x7ff3f80ff070 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3f659c700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 0x7ff3f8103000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00c8f0 tx=0x7ff3ec00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3ec0078e0 con 0x7ff3f80ff070 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f8101fb0 con 0x7ff3f80ff070 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff3ec00f450 con 0x7ff3f80ff070 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3ec018610 con 0x7ff3f80ff070 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.880+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f819da10 con 0x7ff3f80ff070 2024-05-06T14:29:16.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.881+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f8107bc0 con 0x7ff3f80ff070 2024-05-06T14:29:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.882+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ff3ec00f5c0 con 0x7ff3f80ff070 2024-05-06T14:29:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.882+0000 7ff3e77fe700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff3e006c680 0x7ff3e006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.883+0000 7ff3f5d9b700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff3e006c680 0x7ff3e006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.883+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7ff3ec08b8e0 con 0x7ff3f80ff070 2024-05-06T14:29:16.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.883+0000 7ff3f5d9b700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff3e006c680 0x7ff3e006eb30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8006010 tx=0x7ff3e800b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:16.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:16.886+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff3ec059b70 con 0x7ff3f80ff070 2024-05-06T14:29:17.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.127+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7ff3f819dfc0 con 0x7ff3f80ff070 2024-05-06T14:29:17.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.357+0000 7ff3e77fe700 1 -- 172.21.15.153:0/360769680 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff3ec059700 con 0x7ff3f80ff070 2024-05-06T14:29:17.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff3e006c680 msgr2=0x7ff3e006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:17.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff3e006c680 0x7ff3e006eb30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8006010 tx=0x7ff3e800b540 comp rx=0 tx=0).stop 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 msgr2=0x7ff3f8103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 0x7ff3f8103000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00c8f0 tx=0x7ff3ec00cc00 comp rx=0 tx=0).stop 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 shutdown_connections 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3f80ff070 0x7ff3f8103000 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff3e006c680 0x7ff3e006eb30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 --2- 172.21.15.153:0/360769680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100270 0x7ff3f8101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.360+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 >> 172.21.15.153:0/360769680 conn(0x7ff3f80fa600 msgr2=0x7ff3f8068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.361+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 shutdown_connections 2024-05-06T14:29:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:17.361+0000 7ff3fc805700 1 -- 172.21.15.153:0/360769680 wait complete. 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6826/4248251632,v1:172.21.15.153:6827/4248251632] up:active 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: fsmap cephfs:1 {0=cephfs.smithi153.hiwroq=up:active} 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/360769680' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ftbnnk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ftbnnk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:17 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:17.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6826/4248251632,v1:172.21.15.153:6827/4248251632] up:active 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: fsmap cephfs:1 {0=cephfs.smithi153.hiwroq=up:active} 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/360769680' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ftbnnk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.ftbnnk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:17.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:17 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:18.004 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:29:18.015 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:29:18.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: Deploying daemon mds.cephfs.smithi153.ftbnnk on smithi153 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] up:boot 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/360769680' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: daemon mds.cephfs.smithi190.psnzbz assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-06T14:29:18.393 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: fsmap cephfs:1 {0=cephfs.smithi153.hiwroq=up:active} 1 up:standby 2024-05-06T14:29:18.394 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.psnzbz"}]: dispatch 2024-05-06T14:29:18.394 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:creating} 2024-05-06T14:29:18.394 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:18 smithi190 ceph-mon[33671]: daemon mds.cephfs.smithi190.psnzbz is now active in filesystem cephfs as rank 1 2024-05-06T14:29:18.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:29:18.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: Deploying daemon mds.cephfs.smithi153.ftbnnk on smithi153 2024-05-06T14:29:18.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-06T14:29:18.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] up:boot 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/360769680' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: daemon mds.cephfs.smithi190.psnzbz assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: fsmap cephfs:1 {0=cephfs.smithi153.hiwroq=up:active} 1 up:standby 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.psnzbz"}]: dispatch 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:creating} 2024-05-06T14:29:18.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:18 smithi153 ceph-mon[27177]: daemon mds.cephfs.smithi190.psnzbz is now active in filesystem cephfs as rank 1 2024-05-06T14:29:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:19.498 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:19 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] up:active 2024-05-06T14:29:19.498 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:19 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 2024-05-06T14:29:19.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:19 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] up:active 2024-05-06T14:29:19.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:19 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 2024-05-06T14:29:20.492 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:20 smithi153 ceph-mon[27177]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 5 op/s 2024-05-06T14:29:20.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:20 smithi190 ceph-mon[33671]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 5 op/s 2024-05-06T14:29:20.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.729+0000 7f48d3319700 1 -- 172.21.15.153:0/2695781884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 msgr2=0x7f48cc100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:20.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.729+0000 7f48d3319700 1 --2- 172.21.15.153:0/2695781884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc100580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009910 tx=0x7f48c8009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:20.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- 172.21.15.153:0/2695781884 shutdown_connections 2024-05-06T14:29:20.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 --2- 172.21.15.153:0/2695781884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc101370 0x7f48cc1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 --2- 172.21.15.153:0/2695781884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc100580 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- 172.21.15.153:0/2695781884 >> 172.21.15.153:0/2695781884 conn(0x7f48cc0fb6e0 msgr2=0x7f48cc0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- 172.21.15.153:0/2695781884 shutdown_connections 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- 172.21.15.153:0/2695781884 wait complete. 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 Processor -- start 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- start start 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc101370 0x7f48cc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc198700 con 0x7f48cc100170 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d3319700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc198840 con 0x7f48cc101370 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.730+0000 7f48d10b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d10b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46844/0 (socket says 172.21.15.153:46844) 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d10b5700 1 -- 172.21.15.153:0/1060259406 learned_addr learned my addr 172.21.15.153:0/1060259406 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d08b4700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc101370 0x7f48cc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:20.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d10b5700 1 -- 172.21.15.153:0/1060259406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc101370 msgr2=0x7f48cc1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d10b5700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc101370 0x7f48cc1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d10b5700 1 -- 172.21.15.153:0/1060259406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c80095f0 con 0x7f48cc100170 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d10b5700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc197ba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f48c800bda0 tx=0x7f48c800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c801d070 con 0x7f48cc100170 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f48c800f460 con 0x7f48cc100170 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc19d290 con 0x7f48cc100170 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48c8021620 con 0x7f48cc100170 2024-05-06T14:29:20.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.731+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc19d780 con 0x7f48cc100170 2024-05-06T14:29:20.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.732+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc105450 con 0x7f48cc100170 2024-05-06T14:29:20.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.733+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f48c800f5d0 con 0x7f48cc100170 2024-05-06T14:29:20.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.733+0000 7f48c27fc700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f48b806c680 0x7f48b806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:20.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.733+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f48c808cb00 con 0x7f48cc100170 2024-05-06T14:29:20.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.733+0000 7f48d08b4700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f48b806c680 0x7f48b806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:20.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.733+0000 7f48d08b4700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f48b806c680 0x7f48b806eb30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48bc005950 tx=0x7f48bc00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:20.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.739+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f48c805acf0 con 0x7f48cc100170 2024-05-06T14:29:20.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:20.995+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f48cc02cc30 con 0x7f48cc100170 2024-05-06T14:29:21.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.371+0000 7f48c27fc700 1 -- 172.21.15.153:0/1060259406 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48c805a880 con 0x7f48cc100170 2024-05-06T14:29:21.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f48b806c680 msgr2=0x7f48b806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:21.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f48b806c680 0x7f48b806eb30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48bc005950 tx=0x7f48bc00b410 comp rx=0 tx=0).stop 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 msgr2=0x7f48cc197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc197ba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f48c800bda0 tx=0x7f48c800bdd0 comp rx=0 tx=0).stop 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 shutdown_connections 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48cc100170 0x7f48cc197ba0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f48b806c680 0x7f48b806eb30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 --2- 172.21.15.153:0/1060259406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48cc101370 0x7f48cc1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.373+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 >> 172.21.15.153:0/1060259406 conn(0x7f48cc0fb6e0 msgr2=0x7f48cc102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.374+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 shutdown_connections 2024-05-06T14:29:21.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:21.374+0000 7f48d3319700 1 -- 172.21.15.153:0/1060259406 wait complete. 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1060259406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.quxdec", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.quxdec", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:21.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:21 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:21.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1060259406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-05-06T14:29:21.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:21.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:21.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:21.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.quxdec", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-06T14:29:21.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi190.quxdec", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-06T14:29:21.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:21 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:21.964 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:29:21.975 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:29:21.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-05-06T14:29:22.184 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:22.622 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s wr, 8 op/s 2024-05-06T14:29:22.622 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: Deploying daemon mds.cephfs.smithi190.quxdec on smithi190 2024-05-06T14:29:22.622 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-05-06T14:29:22.622 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: Cluster is now healthy 2024-05-06T14:29:22.622 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1060259406' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-05-06T14:29:22.622 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:boot 2024-05-06T14:29:22.623 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 1 up:standby 2024-05-06T14:29:22.623 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:22 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ftbnnk"}]: dispatch 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s wr, 8 op/s 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: Deploying daemon mds.cephfs.smithi190.quxdec on smithi190 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: Cluster is now healthy 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1060259406' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:boot 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 1 up:standby 2024-05-06T14:29:22.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:22 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ftbnnk"}]: dispatch 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 -- 172.21.15.153:0/2170462978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a640ff060 msgr2=0x7f1a640ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/2170462978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a640ff060 0x7f1a640ff470 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009910 tx=0x7f1a58009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 -- 172.21.15.153:0/2170462978 shutdown_connections 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/2170462978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a64100260 0x7f1a641006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/2170462978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a640ff060 0x7f1a640ff470 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 -- 172.21.15.153:0/2170462978 >> 172.21.15.153:0/2170462978 conn(0x7f1a640fa610 msgr2=0x7f1a640fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 -- 172.21.15.153:0/2170462978 shutdown_connections 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.745+0000 7f1a6abf6700 1 -- 172.21.15.153:0/2170462978 wait complete. 2024-05-06T14:29:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a6abf6700 1 Processor -- start 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a6abf6700 1 -- start start 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a6abf6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a640ff060 0x7f1a64102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a6abf6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 0x7f1a64101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a6abf6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64101b80 con 0x7f1a64100260 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a6abf6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64101cc0 con 0x7f1a640ff060 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a63fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 0x7f1a64101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a63fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 0x7f1a64101640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46878/0 (socket says 172.21.15.153:46878) 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a63fff700 1 -- 172.21.15.153:0/580785627 learned_addr learned my addr 172.21.15.153:0/580785627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a68992700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a640ff060 0x7f1a64102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a63fff700 1 -- 172.21.15.153:0/580785627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a640ff060 msgr2=0x7f1a64102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a63fff700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a640ff060 0x7f1a64102ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.746+0000 7f1a63fff700 1 -- 172.21.15.153:0/580785627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580095f0 con 0x7f1a64100260 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.747+0000 7f1a63fff700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 0x7f1a64101640 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000d8d0 tx=0x7f1a5000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:23.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.747+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a500098a0 con 0x7f1a64100260 2024-05-06T14:29:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.747+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1a50010460 con 0x7f1a64100260 2024-05-06T14:29:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.747+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a64101fa0 con 0x7f1a64100260 2024-05-06T14:29:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.747+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a5000f5f0 con 0x7f1a64100260 2024-05-06T14:29:23.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.747+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6419da00 con 0x7f1a64100260 2024-05-06T14:29:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.748+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a64107bb0 con 0x7f1a64100260 2024-05-06T14:29:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.748+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f1a50009a00 con 0x7f1a64100260 2024-05-06T14:29:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.749+0000 7f1a61ffb700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f1a5406c680 0x7f1a5406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.749+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f1a5008b6b0 con 0x7f1a64100260 2024-05-06T14:29:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.749+0000 7f1a68992700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f1a5406c680 0x7f1a5406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:23.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.749+0000 7f1a68992700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f1a5406c680 0x7f1a5406eb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1a58006010 tx=0x7f1a5800b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:23.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.753+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1a500598a0 con 0x7f1a64100260 2024-05-06T14:29:23.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:23.997+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a64107d70 con 0x7f1a64100260 2024-05-06T14:29:24.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.375+0000 7f1a61ffb700 1 -- 172.21.15.153:0/580785627 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7f1a50059430 con 0x7f1a64100260 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f1a5406c680 msgr2=0x7f1a5406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f1a5406c680 0x7f1a5406eb30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1a58006010 tx=0x7f1a5800b540 comp rx=0 tx=0).stop 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 msgr2=0x7f1a64101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 0x7f1a64101640 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000d8d0 tx=0x7f1a5000dbe0 comp rx=0 tx=0).stop 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 shutdown_connections 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f1a5406c680 0x7f1a5406eb30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a640ff060 0x7f1a64102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:24.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 --2- 172.21.15.153:0/580785627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a64100260 0x7f1a64101640 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:24.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 >> 172.21.15.153:0/580785627 conn(0x7f1a640fa610 msgr2=0x7f1a64068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:24.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 shutdown_connections 2024-05-06T14:29:24.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:24.378+0000 7f1a6abf6700 1 -- 172.21.15.153:0/580785627 wait complete. 2024-05-06T14:29:24.367 INFO:teuthology.orchestra.run.smithi153.stderr:inline data enabled 2024-05-06T14:29:24.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/580785627' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-06T14:29:24.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:24 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/580785627' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-06T14:29:24.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:24 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:29:25.070 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:29:25.081 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:29:25.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.190:6826/804379542,v1:172.21.15.190:6827/804379542] up:boot 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/580785627' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.quxdec"}]: dispatch 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:25.681 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.190:6826/804379542,v1:172.21.15.190:6827/804379542] up:boot 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/580785627' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.quxdec"}]: dispatch 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:25.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:25.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:25.830 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:standby 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] up:active 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: Dropping low affinity active daemon mds.cephfs.smithi153.hiwroq in favor of higher affinity standby. 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: Replacing daemon mds.cephfs.smithi153.hiwroq as rank 0 with standby daemon mds.cephfs.smithi153.ftbnnk 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: osdmap e47: 6 total, 6 up, 6 in 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:replay,1=cephfs.smithi190.psnzbz=up:active} 1 up:standby 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:26 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:26.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-05-06T14:29:26.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:standby 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] up:active 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: Dropping low affinity active daemon mds.cephfs.smithi153.hiwroq in favor of higher affinity standby. 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: Replacing daemon mds.cephfs.smithi153.hiwroq as rank 0 with standby daemon mds.cephfs.smithi153.ftbnnk 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.hiwroq=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: osdmap e47: 6 total, 6 up, 6 in 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:replay,1=cephfs.smithi190.psnzbz=up:active} 1 up:standby 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:26.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:26 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:27.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 -- 172.21.15.153:0/1134559959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0100190 msgr2=0x7f63c01005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 --2- 172.21.15.153:0/1134559959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0100190 0x7f63c01005a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b0a0 tx=0x7f63b800b3b0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 -- 172.21.15.153:0/1134559959 shutdown_connections 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 --2- 172.21.15.153:0/1134559959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63c0101390 0x7f63c01017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 --2- 172.21.15.153:0/1134559959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0100190 0x7f63c01005a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 -- 172.21.15.153:0/1134559959 >> 172.21.15.153:0/1134559959 conn(0x7f63c00fb720 msgr2=0x7f63c00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 -- 172.21.15.153:0/1134559959 shutdown_connections 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.452+0000 7f63c7549700 1 -- 172.21.15.153:0/1134559959 wait complete. 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c7549700 1 Processor -- start 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c7549700 1 -- start start 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c7549700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63c0100190 0x7f63c0074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c7549700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 0x7f63c0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c7549700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c0073680 con 0x7f63c0101390 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c7549700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c00737c0 con 0x7f63c0100190 2024-05-06T14:29:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c4ae4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 0x7f63c0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c4ae4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 0x7f63c0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46922/0 (socket says 172.21.15.153:46922) 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c4ae4700 1 -- 172.21.15.153:0/221110775 learned_addr learned my addr 172.21.15.153:0/221110775 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c52e5700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63c0100190 0x7f63c0074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c4ae4700 1 -- 172.21.15.153:0/221110775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63c0100190 msgr2=0x7f63c0074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c4ae4700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63c0100190 0x7f63c0074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.453+0000 7f63c4ae4700 1 -- 172.21.15.153:0/221110775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8009d00 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.454+0000 7f63c4ae4700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 0x7f63c0073140 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f63b400b4b0 tx=0x7f63b400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.454+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63b4011800 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.454+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c0073aa0 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.454+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f63b4011e10 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.454+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63b400f2e0 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.454+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c019db70 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.455+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c0108d60 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.455+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f63b4010380 con 0x7f63c0101390 2024-05-06T14:29:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.456+0000 7f63be7fc700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f63a806c680 0x7f63a806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.456+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f63b408b4b0 con 0x7f63c0101390 2024-05-06T14:29:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.456+0000 7f63c52e5700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f63a806c680 0x7f63a806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.456+0000 7f63c52e5700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f63a806c680 0x7f63a806eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009fd0 tx=0x7f63b8000f40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.460+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f63b40595c0 con 0x7f63c0101390 2024-05-06T14:29:27.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.714+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f63c019e120 con 0x7f63c0101390 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.715+0000 7f63be7fc700 1 -- 172.21.15.153:0/221110775 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1824 (secure 0 0 0) 0x7f63b4059150 con 0x7f63c0101390 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:e13 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 13 2024-05-06T14:29:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-05-06T14:29:10.325593+0000 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-05-06T14:29:27.022747+0000 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 47 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14494,1=24271} 2024-05-06T14:29:27.704 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data enabled 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.ftbnnk{0:14494} state up:resolve seq 3 join_fscid=1 addr [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] compat {c=[1],r=[1],i=[7ff]}] 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi190.psnzbz{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.190:6824/3762223844,v1:172.21.15.190:6825/3762223844] compat {c=[1],r=[1],i=[7ff]}] 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.hiwroq{-1:14506} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/4261352408,v1:172.21.15.153:6827/4261352408] compat {c=[1],r=[1],i=[7ff]}] 2024-05-06T14:29:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi190.quxdec{-1:24287} state up:standby seq 1 addr [v2:172.21.15.190:6826/804379542,v1:172.21.15.190:6827/804379542] compat {c=[1],r=[1],i=[7ff]}] 2024-05-06T14:29:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f63a806c680 msgr2=0x7f63a806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f63a806c680 0x7f63a806eb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009fd0 tx=0x7f63b8000f40 comp rx=0 tx=0).stop 2024-05-06T14:29:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 msgr2=0x7f63c0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 0x7f63c0073140 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f63b400b4b0 tx=0x7f63b400b7c0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 shutdown_connections 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f63a806c680 0x7f63a806eb30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63c0100190 0x7f63c0074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 --2- 172.21.15.153:0/221110775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c0101390 0x7f63c0073140 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 >> 172.21.15.153:0/221110775 conn(0x7f63c00fb720 msgr2=0x7f63c01045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 shutdown_connections 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:27.718+0000 7f63c7549700 1 -- 172.21.15.153:0/221110775 wait complete. 2024-05-06T14:29:27.707 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 13 2024-05-06T14:29:28.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:28 smithi153 ceph-mon[27177]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-05-06T14:29:28.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:28 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:resolve 2024-05-06T14:29:28.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:28 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6826/4261352408,v1:172.21.15.153:6827/4261352408] up:boot 2024-05-06T14:29:28.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:28 smithi153 ceph-mon[27177]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:resolve,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:28.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.hiwroq"}]: dispatch 2024-05-06T14:29:28.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:28 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/221110775' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-06T14:29:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:28 smithi190 ceph-mon[33671]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 8 op/s 2024-05-06T14:29:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:28 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:resolve 2024-05-06T14:29:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:28 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6826/4261352408,v1:172.21.15.153:6827/4261352408] up:boot 2024-05-06T14:29:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:28 smithi190 ceph-mon[33671]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:resolve,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.hiwroq"}]: dispatch 2024-05-06T14:29:28.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:28 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/221110775' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-06T14:29:28.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-05-06T14:29:29.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:29 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:reconnect 2024-05-06T14:29:29.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:29 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.190:6826/804379542,v1:172.21.15.190:6827/804379542] up:standby 2024-05-06T14:29:29.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:29 smithi153 ceph-mon[27177]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:reconnect,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:29 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:reconnect 2024-05-06T14:29:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:29 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.190:6826/804379542,v1:172.21.15.190:6827/804379542] up:standby 2024-05-06T14:29:29.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:29 smithi190 ceph-mon[33671]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:reconnect,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:29.670 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:30.125 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:30 smithi153 ceph-mon[27177]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1.6 KiB/s wr, 5 op/s 2024-05-06T14:29:30.125 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:30 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:rejoin 2024-05-06T14:29:30.125 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:30 smithi153 ceph-mon[27177]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:rejoin,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:30.125 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:30 smithi153 ceph-mon[27177]: daemon mds.cephfs.smithi153.ftbnnk is now active in filesystem cephfs as rank 0 2024-05-06T14:29:30.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:30 smithi190 ceph-mon[33671]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1.6 KiB/s wr, 5 op/s 2024-05-06T14:29:30.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:30 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:rejoin 2024-05-06T14:29:30.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:30 smithi190 ceph-mon[33671]: fsmap cephfs:2/2 {0=cephfs.smithi153.ftbnnk=up:rejoin,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:30.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:30 smithi190 ceph-mon[33671]: daemon mds.cephfs.smithi153.ftbnnk is now active in filesystem cephfs as rank 0 2024-05-06T14:29:31.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:31 smithi153 ceph-mon[27177]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-06T14:29:31.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:31 smithi153 ceph-mon[27177]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:active 2024-05-06T14:29:31.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:31 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.ftbnnk=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:31.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:31 smithi190 ceph-mon[33671]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-06T14:29:31.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:31 smithi190 ceph-mon[33671]: mds.? [v2:172.21.15.153:6828/179559541,v1:172.21.15.153:6829/179559541] up:active 2024-05-06T14:29:31.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:31 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.ftbnnk=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 -- 172.21.15.153:0/3638425827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 msgr2=0x7fbec41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/3638425827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec41006a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009910 tx=0x7fbeb4009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 -- 172.21.15.153:0/3638425827 shutdown_connections 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/3638425827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec41006a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/3638425827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec40ff050 0x7fbec40ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 -- 172.21.15.153:0/3638425827 >> 172.21.15.153:0/3638425827 conn(0x7fbec40fa600 msgr2=0x7fbec40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 -- 172.21.15.153:0/3638425827 shutdown_connections 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.988+0000 7fbecc6c9700 1 -- 172.21.15.153:0/3638425827 wait complete. 2024-05-06T14:29:31.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbecc6c9700 1 Processor -- start 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbecc6c9700 1 -- start start 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbecc6c9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec40ff050 0x7fbec4193750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbecc6c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec4193c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbecc6c9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec41942b0 con 0x7fbec4100250 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbecc6c9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec41943f0 con 0x7fbec40ff050 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec4193c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec4193c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49496/0 (socket says 172.21.15.153:49496) 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbeca465700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec40ff050 0x7fbec4193750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 -- 172.21.15.153:0/2155704160 learned_addr learned my addr 172.21.15.153:0/2155704160 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 -- 172.21.15.153:0/2155704160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec40ff050 msgr2=0x7fbec4193750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec40ff050 0x7fbec4193750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 -- 172.21.15.153:0/2155704160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40095f0 con 0x7fbec4100250 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.989+0000 7fbec9c64700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec4193c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4000c00 tx=0x7fbeb400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:31.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.990+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb401d070 con 0x7fbec4100250 2024-05-06T14:29:31.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.990+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4198e40 con 0x7fbec4100250 2024-05-06T14:29:31.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.990+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbeb4005dc0 con 0x7fbec4100250 2024-05-06T14:29:31.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.990+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb40226a0 con 0x7fbec4100250 2024-05-06T14:29:31.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.990+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec4199300 con 0x7fbec4100250 2024-05-06T14:29:31.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.991+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec4106a90 con 0x7fbec4100250 2024-05-06T14:29:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.991+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fbeb40038f0 con 0x7fbec4100250 2024-05-06T14:29:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.992+0000 7fbebb7fe700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbeb006c680 0x7fbeb006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.992+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbeb408cdc0 con 0x7fbec4100250 2024-05-06T14:29:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.992+0000 7fbeca465700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbeb006c680 0x7fbeb006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.992+0000 7fbeca465700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbeb006c680 0x7fbeb006eb30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbec41000b0 tx=0x7fbec000d040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:31.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:31.996+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbeb405aed0 con 0x7fbec4100250 2024-05-06T14:29:32.142 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:32 smithi153 ceph-mon[27177]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-05-06T14:29:32.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.237+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbec4106c50 con 0x7fbec4100250 2024-05-06T14:29:32.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.238+0000 7fbebb7fe700 1 -- 172.21.15.153:0/2155704160 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4760 (secure 0 0 0) 0x7fbeb4003680 con 0x7fbec4100250 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbeb006c680 msgr2=0x7fbeb006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbeb006c680 0x7fbeb006eb30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbec41000b0 tx=0x7fbec000d040 comp rx=0 tx=0).stop 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 msgr2=0x7fbec4193c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec4193c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4000c00 tx=0x7fbeb400f690 comp rx=0 tx=0).stop 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 shutdown_connections 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbeb006c680 0x7fbeb006eb30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbec40ff050 0x7fbec4193750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 --2- 172.21.15.153:0/2155704160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec4100250 0x7fbec4193c90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 >> 172.21.15.153:0/2155704160 conn(0x7fbec40fa600 msgr2=0x7fbec4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 shutdown_connections 2024-05-06T14:29:32.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:32.240+0000 7fbecc6c9700 1 -- 172.21.15.153:0/2155704160 wait complete. 2024-05-06T14:29:32.229 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 16 2024-05-06T14:29:32.238 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-05-06T14:29:32.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:32 smithi190 ceph-mon[33671]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-05-06T14:29:32.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-05-06T14:29:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:33 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2155704160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-06T14:29:33.333 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:33.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:33 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2155704160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-06T14:29:34.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:34 smithi190 ceph-mon[33671]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-06T14:29:34.479 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:34 smithi153 ceph-mon[27177]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-06T14:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 -- 172.21.15.153:0/988236570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070100170 msgr2=0x7fa070100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 --2- 172.21.15.153:0/988236570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070100170 0x7fa070100580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa060009910 tx=0x7fa060009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 -- 172.21.15.153:0/988236570 shutdown_connections 2024-05-06T14:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 --2- 172.21.15.153:0/988236570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070101370 0x7fa0701017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:34.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 --2- 172.21.15.153:0/988236570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070100170 0x7fa070100580 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 -- 172.21.15.153:0/988236570 >> 172.21.15.153:0/988236570 conn(0x7fa0700fb6e0 msgr2=0x7fa0700fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 -- 172.21.15.153:0/988236570 shutdown_connections 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.001+0000 7fa075283700 1 -- 172.21.15.153:0/988236570 wait complete. 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa075283700 1 Processor -- start 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa075283700 1 -- start start 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa075283700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 0x7fa070195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa075283700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 0x7fa070195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa075283700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0701964f0 con 0x7fa070101370 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa075283700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070196630 con 0x7fa070100170 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa06e7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 0x7fa070195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa06effd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 0x7fa070195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa06effd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 0x7fa070195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:38648/0 (socket says 172.21.15.153:38648) 2024-05-06T14:29:34.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.002+0000 7fa06e7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 0x7fa070195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49510/0 (socket says 172.21.15.153:49510) 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa06effd700 1 -- 172.21.15.153:0/2842648342 learned_addr learned my addr 172.21.15.153:0/2842648342 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa06e7fc700 1 -- 172.21.15.153:0/2842648342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 msgr2=0x7fa070195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa06e7fc700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 0x7fa070195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa06e7fc700 1 -- 172.21.15.153:0/2842648342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600095f0 con 0x7fa070101370 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa06effd700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 0x7fa070195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa06e7fc700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 0x7fa070195ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e970 tx=0x7fa06400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa06400c9d0 con 0x7fa070101370 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07019b0e0 con 0x7fa070101370 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa064010430 con 0x7fa070101370 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa064018650 con 0x7fa070101370 2024-05-06T14:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.003+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07019b630 con 0x7fa070101370 2024-05-06T14:29:34.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.004+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa070108cc0 con 0x7fa070101370 2024-05-06T14:29:34.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.005+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa06400cb30 con 0x7fa070101370 2024-05-06T14:29:34.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.005+0000 7fa057fff700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa05806c680 0x7fa05806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:34.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.005+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa064014070 con 0x7fa070101370 2024-05-06T14:29:34.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.005+0000 7fa06effd700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa05806c680 0x7fa05806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:34.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.006+0000 7fa06effd700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa05806c680 0x7fa05806eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa060009fd0 tx=0x7fa060005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:34.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.011+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa064051720 con 0x7fa070101370 2024-05-06T14:29:35.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.309+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fa070108ee0 con 0x7fa070101370 2024-05-06T14:29:35.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.309+0000 7fa057fff700 1 -- 172.21.15.153:0/2842648342 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+83 (secure 0 0 0) 0x7fa064059d60 con 0x7fa070101370 2024-05-06T14:29:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa05806c680 msgr2=0x7fa05806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa05806c680 0x7fa05806eb30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa060009fd0 tx=0x7fa060005940 comp rx=0 tx=0).stop 2024-05-06T14:29:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 msgr2=0x7fa070195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 0x7fa070195ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e970 tx=0x7fa06400ec80 comp rx=0 tx=0).stop 2024-05-06T14:29:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 shutdown_connections 2024-05-06T14:29:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa05806c680 0x7fa05806eb30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:35.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa070100170 0x7fa070195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:35.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 --2- 172.21.15.153:0/2842648342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa070101370 0x7fa070195ed0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:35.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 >> 172.21.15.153:0/2842648342 conn(0x7fa0700fb6e0 msgr2=0x7fa0701045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:35.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 shutdown_connections 2024-05-06T14:29:35.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:35.311+0000 7fa075283700 1 -- 172.21.15.153:0/2842648342 wait complete. 2024-05-06T14:29:35.308 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-05-06T14:29:35.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-06T14:29:35.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:35 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2842648342' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-05-06T14:29:36.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-06T14:29:36.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:35 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2842648342' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-05-06T14:29:36.468 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-05-06T14:29:36.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T14:29:36.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2603297471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 msgr2=0x7ffa100726f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2603297471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa100726f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009880 tx=0x7ffa0002bce0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2603297471 shutdown_connections 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2603297471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa10072c30 0x7ffa100730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2603297471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa100726f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2603297471 >> 172.21.15.153:0/2603297471 conn(0x7ffa100faae0 msgr2=0x7ffa100fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2603297471 shutdown_connections 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2603297471 wait complete. 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.627+0000 7ffa14ba2640 1 Processor -- start 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa14ba2640 1 -- start start 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa14ba2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa10072c30 0x7ffa10196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa14ba2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa10197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:36.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa14ba2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10197a20 con 0x7ffa10074200 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa14ba2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10197b90 con 0x7ffa10072c30 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa10197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa10197450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49536/0 (socket says 172.21.15.153:49536) 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 -- 172.21.15.153:0/2314244791 learned_addr learned my addr 172.21.15.153:0/2314244791 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0f7fe640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa10072c30 0x7ffa10196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 -- 172.21.15.153:0/2314244791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa10072c30 msgr2=0x7ffa10196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa10072c30 0x7ffa10196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 -- 172.21.15.153:0/2314244791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00009560 con 0x7ffa10074200 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.628+0000 7ffa0effd640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa10197450 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c8f0 tx=0x7ffa0400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.629+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa04007b80 con 0x7ffa10074200 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.629+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1019c620 con 0x7ffa10074200 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.629+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffa04007ce0 con 0x7ffa10074200 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.629+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0400f450 con 0x7ffa10074200 2024-05-06T14:29:36.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.629+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1019cb80 con 0x7ffa10074200 2024-05-06T14:29:36.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.630+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1010a350 con 0x7ffa10074200 2024-05-06T14:29:36.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.630+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7ffa04004f60 con 0x7ffa10074200 2024-05-06T14:29:36.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.631+0000 7ffa0cff9640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff9f806c6b0 0x7ff9f806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:36.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.631+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffa0408b650 con 0x7ffa10074200 2024-05-06T14:29:36.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.631+0000 7ffa0f7fe640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff9f806c6b0 0x7ff9f806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:36.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.631+0000 7ffa0f7fe640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff9f806c6b0 0x7ff9f806eb60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa00000c00 tx=0x7ffa00005fd0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:36.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.635+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ffa04055d80 con 0x7ffa10074200 2024-05-06T14:29:36.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.810+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa1019ce00 con 0x7ffa10074200 2024-05-06T14:29:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.811+0000 7ffa0cff9640 1 -- 172.21.15.153:0/2314244791 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15159 (secure 0 0 0) 0x7ffa040593a0 con 0x7ffa10074200 2024-05-06T14:29:36.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:36.798 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":47,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","created":"2024-05-06T14:22:12.905455+0000","modified":"2024-05-06T14:29:26.017181+0000","last_up_change":"2024-05-06T14:27:54.584488+0000","last_in_change":"2024-05-06T14:27:35.267866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T14:26:41.240977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-05-06T14:29:08.798039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-05-06T14:29:09.557951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de83eb29-8176-4eb5-9be6-64b9e5449892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6803","nonce":908509767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6805","nonce":908509767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6809","nonce":908509767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6807","nonce":908509767}]},"public_addr":"172.21.15.153:6803/908509767","cluster_addr":"172.21.15.153:6805/908509767","heartbeat_back_addr":"172.21.15.153:6809/908509767","heartbeat_front_addr":"172.21.15.153:6807/908509767","state":["exists","up"]},{"osd":1,"uuid":"24a1c7ac-a168-4e7d-8c34-3e4c73408756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6811","nonce":3716718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6813","nonce":3716718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6817","nonce":3716718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6815","nonce":3716718209}]},"public_addr":"172.21.15.153:6811/3716718209","cluster_addr":"172.21.15.153:6813/3716718209","heartbeat_back_addr":"172.21.15.153:6817/3716718209","heartbeat_front_addr":"172.21.15.153:6815/3716718209","state":["exists","up"]},{"osd":2,"uuid":"5b89edb2-1c0e-4481-91ed-157d767c0c05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6819","nonce":3169611803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6821","nonce":3169611803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6825","nonce":3169611803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6823","nonce":3169611803}]},"public_addr":"172.21.15.153:6819/3169611803","cluster_addr":"172.21.15.153:6821/3169611803","heartbeat_back_addr":"172.21.15.153:6825/3169611803","heartbeat_front_addr":"172.21.15.153:6823/3169611803","state":["exists","up"]},{"osd":3,"uuid":"74b7823e-2b78-4570-9b7a-89abc9d7fbb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6801","nonce":492034660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6803","nonce":492034660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6807","nonce":492034660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6805","nonce":492034660}]},"public_addr":"172.21.15.190:6801/492034660","cluster_addr":"172.21.15.190:6803/492034660","heartbeat_back_addr":"172.21.15.190:6807/492034660","heartbeat_front_addr":"172.21.15.190:6805/492034660","state":["exists","up"]},{"osd":4,"uuid":"01290c44-3bf1-4da3-bac3-4dff2453cedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6809","nonce":1342651346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6811","nonce":1342651346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6815","nonce":1342651346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6813","nonce":1342651346}]},"public_addr":"172.21.15.190:6809/1342651346","cluster_addr":"172.21.15.190:6811/1342651346","heartbeat_back_addr":"172.21.15.190:6815/1342651346","heartbeat_front_addr":"172.21.15.190:6813/1342651346","state":["exists","up"]},{"osd":5,"uuid":"be366c2e-63d4-4496-b401-772dbe1a2444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6817","nonce":452463735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6819","nonce":452463735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6823","nonce":452463735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6821","nonce":452463735}]},"public_addr":"172.21.15.190:6817/452463735","cluster_addr":"172.21.15.190:6819/452463735","heartbeat_back_addr":"172.21.15.190:6823/452463735","heartbeat_front_addr":"172.21.15.190:6821/452463735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:25:42.510404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:11.276096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:37.855142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:01.189580+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:26.756150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:52.720929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6827/4248251632":"2024-05-07T14:29:26.017160+0000","172.21.15.153:0/3158057303":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6826/4248251632":"2024-05-07T14:29:26.017160+0000","172.21.15.153:6801/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3536681135":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1199411437":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6800/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1309940234":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2948592992":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6801/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/1311303452":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2161963253":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6801/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1460873727":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2433042994":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6800/2228103355":"2024-05-07T14:24:24.923607+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T14:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff9f806c6b0 msgr2=0x7ff9f806eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff9f806c6b0 0x7ff9f806eb60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa00000c00 tx=0x7ffa00005fd0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 msgr2=0x7ffa10197450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:36.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa10197450 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c8f0 tx=0x7ffa0400cdb0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 shutdown_connections 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7ff9f806c6b0 0x7ff9f806eb60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa10074200 0x7ffa10197450 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 --2- 172.21.15.153:0/2314244791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa10072c30 0x7ffa10196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 >> 172.21.15.153:0/2314244791 conn(0x7ffa100faae0 msgr2=0x7ffa10105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 shutdown_connections 2024-05-06T14:29:36.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.813+0000 7ffa14ba2640 1 -- 172.21.15.153:0/2314244791 wait complete. 2024-05-06T14:29:36.813 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T14:26:41.240977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-05-06T14:29:08.798039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-05-06T14:29:09.557951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-06T14:29:36.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-06T14:29:36.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 -- 172.21.15.153:0/927634892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 msgr2=0x7f0488104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:36.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 --2- 172.21.15.153:0/927634892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f0488104dc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f047c0098b0 tx=0x7f047c02bcd0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 -- 172.21.15.153:0/927634892 shutdown_connections 2024-05-06T14:29:36.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 --2- 172.21.15.153:0/927634892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f0488104dc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 --2- 172.21.15.153:0/927634892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04881000c0 0x7f04881024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 -- 172.21.15.153:0/927634892 >> 172.21.15.153:0/927634892 conn(0x7f04880f9a80 msgr2=0x7f04880fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:36.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 -- 172.21.15.153:0/927634892 shutdown_connections 2024-05-06T14:29:36.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 -- 172.21.15.153:0/927634892 wait complete. 2024-05-06T14:29:36.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.934+0000 7f048dd71640 1 Processor -- start 2024-05-06T14:29:36.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f048dd71640 1 -- start start 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f048dd71640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04881000c0 0x7f0488194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f048dd71640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f04881954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f048dd71640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488195ac0 con 0x7f04881029e0 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f048dd71640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488195c30 con 0x7f04881000c0 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f04881954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f04881954f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49552/0 (socket says 172.21.15.153:49552) 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 -- 172.21.15.153:0/1344637580 learned_addr learned my addr 172.21.15.153:0/1344637580 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f04877fe640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04881000c0 0x7f0488194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 -- 172.21.15.153:0/1344637580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04881000c0 msgr2=0x7f0488194fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04881000c0 0x7f0488194fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 -- 172.21.15.153:0/1344637580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c009560 con 0x7f04881029e0 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.935+0000 7f0486ffd640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f04881954f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f047c000c00 tx=0x7f047c004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:36.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.936+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f047c005000 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.936+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048819a660 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.936+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f047c005610 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.936+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f047c034670 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.936+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048819ab60 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.937+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048818f050 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.937+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f047c0035f0 con 0x7f04881029e0 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.938+0000 7f0484ff9640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f047006c5f0 0x7f047006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:36.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.938+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f047c0adce0 con 0x7f04881029e0 2024-05-06T14:29:36.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.938+0000 7f04877fe640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f047006c5f0 0x7f047006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:36.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.938+0000 7f04877fe640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f047006c5f0 0x7f047006eaa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0478005d00 tx=0x7f047800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:36.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:36.942+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f047c078410 con 0x7f04881029e0 2024-05-06T14:29:37.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.114+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f048802c520 con 0x7f04881029e0 2024-05-06T14:29:37.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.115+0000 7f0484ff9640 1 -- 172.21.15.153:0/1344637580 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f047c04b020 con 0x7f04881029e0 2024-05-06T14:29:37.102 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-05-06T14:29:37.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.116+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f047006c5f0 msgr2=0x7f047006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.116+0000 7f048dd71640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f047006c5f0 0x7f047006eaa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0478005d00 tx=0x7f047800b410 comp rx=0 tx=0).stop 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.116+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 msgr2=0x7f04881954f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.116+0000 7f048dd71640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f04881954f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f047c000c00 tx=0x7f047c004e30 comp rx=0 tx=0).stop 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.116+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 shutdown_connections 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.117+0000 7f048dd71640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f047006c5f0 0x7f047006eaa0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.117+0000 7f048dd71640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04881029e0 0x7f04881954f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.117+0000 7f048dd71640 1 --2- 172.21.15.153:0/1344637580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04881000c0 0x7f0488194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.117+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 >> 172.21.15.153:0/1344637580 conn(0x7f04880f9a80 msgr2=0x7f04880fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.117+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 shutdown_connections 2024-05-06T14:29:37.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.117+0000 7f048dd71640 1 -- 172.21.15.153:0/1344637580 wait complete. 2024-05-06T14:29:37.110 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:36 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2314244791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:29:37.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-05-06T14:29:37.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:36 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2314244791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 -- 172.21.15.153:0/1036926835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00fe570 msgr2=0x7f7ef00fe980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 --2- 172.21.15.153:0/1036926835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00fe570 0x7f7ef00fe980 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009880 tx=0x7f7ee002bce0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 -- 172.21.15.153:0/1036926835 shutdown_connections 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 --2- 172.21.15.153:0/1036926835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef00ff760 0x7f7ef00ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 --2- 172.21.15.153:0/1036926835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00fe570 0x7f7ef00fe980 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 -- 172.21.15.153:0/1036926835 >> 172.21.15.153:0/1036926835 conn(0x7f7ef00f9ad0 msgr2=0x7f7ef00fbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:37.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 -- 172.21.15.153:0/1036926835 shutdown_connections 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.230+0000 7f7ef5358640 1 -- 172.21.15.153:0/1036926835 wait complete. 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ef5358640 1 Processor -- start 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ef5358640 1 -- start start 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ef5358640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef00fe570 0x7f7ef0068ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ef5358640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 0x7f7ef0199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ef5358640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef019a290 con 0x7f7ef00ff760 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ef5358640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef019a3d0 con 0x7f7ef00fe570 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 0x7f7ef0199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 0x7f7ef0199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49562/0 (socket says 172.21.15.153:49562) 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 -- 172.21.15.153:0/3691994962 learned_addr learned my addr 172.21.15.153:0/3691994962 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7eeeffd640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef00fe570 0x7f7ef0068ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 -- 172.21.15.153:0/3691994962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef00fe570 msgr2=0x7f7ef0068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef00fe570 0x7f7ef0068ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 -- 172.21.15.153:0/3691994962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee0009560 con 0x7f7ef00ff760 2024-05-06T14:29:37.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.231+0000 7f7ee7fff640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 0x7f7ef0199cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800e970 tx=0x7f7ed800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:37.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.232+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed800cc90 con 0x7f7ef00ff760 2024-05-06T14:29:37.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.232+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ed800cdf0 con 0x7f7ef00ff760 2024-05-06T14:29:37.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.232+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed8010430 con 0x7f7ef00ff760 2024-05-06T14:29:37.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.232+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef019a6b0 con 0x7f7ef00ff760 2024-05-06T14:29:37.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.232+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef019abc0 con 0x7f7ef00ff760 2024-05-06T14:29:37.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.233+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef01072a0 con 0x7f7ef00ff760 2024-05-06T14:29:37.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.233+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f7ed8004f40 con 0x7f7ef00ff760 2024-05-06T14:29:37.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.234+0000 7f7eecff9640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7edc06c5f0 0x7f7edc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.234+0000 7f7eeeffd640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7edc06c5f0 0x7f7edc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.234+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7ed8014070 con 0x7f7ef00ff760 2024-05-06T14:29:37.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.234+0000 7f7eeeffd640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7edc06c5f0 0x7f7edc06eaa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0005fd0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:37.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.238+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7ed8056520 con 0x7f7ef00ff760 2024-05-06T14:29:37.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.419+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f7ef019b090 con 0x7f7ef00ff760 2024-05-06T14:29:37.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.419+0000 7f7eecff9640 1 -- 172.21.15.153:0/3691994962 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7ed8059b40 con 0x7f7ef00ff760 2024-05-06T14:29:37.406 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-05-06T14:29:37.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7edc06c5f0 msgr2=0x7f7edc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7edc06c5f0 0x7f7edc06eaa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0000c00 tx=0x7f7ee0005fd0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 msgr2=0x7f7ef0199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 0x7f7ef0199cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800e970 tx=0x7f7ed800ed70 comp rx=0 tx=0).stop 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 shutdown_connections 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f7edc06c5f0 0x7f7edc06eaa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ef00ff760 0x7f7ef0199cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 --2- 172.21.15.153:0/3691994962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ef00fe570 0x7f7ef0068ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 >> 172.21.15.153:0/3691994962 conn(0x7f7ef00f9ad0 msgr2=0x7f7ef0102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 shutdown_connections 2024-05-06T14:29:37.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.422+0000 7f7ef5358640 1 -- 172.21.15.153:0/3691994962 wait complete. 2024-05-06T14:29:37.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 -- 172.21.15.153:0/412265394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 msgr2=0x7fede4100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 --2- 172.21.15.153:0/412265394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4100cb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009880 tx=0x7fedd802bce0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 -- 172.21.15.153:0/412265394 shutdown_connections 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 --2- 172.21.15.153:0/412265394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4100cb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 --2- 172.21.15.153:0/412265394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fede40ff650 0x7fede40ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 -- 172.21.15.153:0/412265394 >> 172.21.15.153:0/412265394 conn(0x7fede40fabd0 msgr2=0x7fede40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 -- 172.21.15.153:0/412265394 shutdown_connections 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 -- 172.21.15.153:0/412265394 wait complete. 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 Processor -- start 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.536+0000 7fedea316640 1 -- start start 2024-05-06T14:29:37.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fedea316640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fede40ff650 0x7fede4194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fedea316640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fedea316640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede4195a60 con 0x7fede4100840 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fedea316640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede4195bd0 con 0x7fede40ff650 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49586/0 (socket says 172.21.15.153:49586) 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 -- 172.21.15.153:0/2855295094 learned_addr learned my addr 172.21.15.153:0/2855295094 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede3fff640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fede40ff650 0x7fede4194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 -- 172.21.15.153:0/2855295094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fede40ff650 msgr2=0x7fede4194f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fede40ff650 0x7fede4194f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 -- 172.21.15.153:0/2855295094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8009560 con 0x7fede4100840 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede37fe640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4195490 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009230 tx=0x7fedd802cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.537+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd8004420 con 0x7fede4100840 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.538+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede419a600 con 0x7fede4100840 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.538+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fedd80054b0 con 0x7fede4100840 2024-05-06T14:29:37.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.538+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede419ab00 con 0x7fede4100840 2024-05-06T14:29:37.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.538+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd8034440 con 0x7fede4100840 2024-05-06T14:29:37.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.539+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede4108390 con 0x7fede4100840 2024-05-06T14:29:37.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.539+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fedd80037d0 con 0x7fede4100840 2024-05-06T14:29:37.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.539+0000 7fede17fa640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fedcc06c5f0 0x7fedcc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.539+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fedd80add20 con 0x7fede4100840 2024-05-06T14:29:37.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.539+0000 7fede3fff640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fedcc06c5f0 0x7fedcc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.540+0000 7fede3fff640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fedcc06c5f0 0x7fedcc06eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fede41006a0 tx=0x7fedd4005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:37.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.544+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fedd8078450 con 0x7fede4100840 2024-05-06T14:29:37.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.715+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fede419ae30 con 0x7fede4100840 2024-05-06T14:29:37.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.715+0000 7fede17fa640 1 -- 172.21.15.153:0/2855295094 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fedd804b020 con 0x7fede4100840 2024-05-06T14:29:37.702 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-05-06T14:29:37.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.717+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fedcc06c5f0 msgr2=0x7fedcc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.717+0000 7fedea316640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fedcc06c5f0 0x7fedcc06eaa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fede41006a0 tx=0x7fedd4005f90 comp rx=0 tx=0).stop 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.717+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 msgr2=0x7fede4195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.717+0000 7fedea316640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4195490 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009230 tx=0x7fedd802cec0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 shutdown_connections 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fedcc06c5f0 0x7fedcc06eaa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fede4100840 0x7fede4195490 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 --2- 172.21.15.153:0/2855295094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fede40ff650 0x7fede4194f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 >> 172.21.15.153:0/2855295094 conn(0x7fede40fabd0 msgr2=0x7fede4103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 shutdown_connections 2024-05-06T14:29:37.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.718+0000 7fedea316640 1 -- 172.21.15.153:0/2855295094 wait complete. 2024-05-06T14:29:37.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-06T14:29:37.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.827+0000 7fdedf80b640 1 -- 172.21.15.153:0/3602024426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80fe590 msgr2=0x7fded80fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.827+0000 7fdedf80b640 1 --2- 172.21.15.153:0/3602024426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80fe590 0x7fded80fe9a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fded0009c20 tx=0x7fded002c1b0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.827+0000 7fdedf80b640 1 -- 172.21.15.153:0/3602024426 shutdown_connections 2024-05-06T14:29:37.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 --2- 172.21.15.153:0/3602024426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fded80ff780 0x7fded80ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 --2- 172.21.15.153:0/3602024426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80fe590 0x7fded80fe9a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 -- 172.21.15.153:0/3602024426 >> 172.21.15.153:0/3602024426 conn(0x7fded80f9b30 msgr2=0x7fded80fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 -- 172.21.15.153:0/3602024426 shutdown_connections 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 -- 172.21.15.153:0/3602024426 wait complete. 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 Processor -- start 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 -- start start 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fded80fe590 0x7fded8194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 0x7fded8195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded8195a60 con 0x7fded80ff780 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedf80b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded8195bd0 con 0x7fded80fe590 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedcda6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 0x7fded8195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedcda6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 0x7fded8195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49608/0 (socket says 172.21.15.153:49608) 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.828+0000 7fdedcda6640 1 -- 172.21.15.153:0/4021000931 learned_addr learned my addr 172.21.15.153:0/4021000931 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedd5a7640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fded80fe590 0x7fded8194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedcda6640 1 -- 172.21.15.153:0/4021000931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fded80fe590 msgr2=0x7fded8194f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedcda6640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fded80fe590 0x7fded8194f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedcda6640 1 -- 172.21.15.153:0/4021000931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded0009900 con 0x7fded80ff780 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedcda6640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 0x7fded8195490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdec800e970 tx=0x7fdec800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdec800cc90 con 0x7fded80ff780 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdec800cdf0 con 0x7fded80ff780 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded819a660 con 0x7fded80ff780 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdec8010430 con 0x7fded80ff780 2024-05-06T14:29:37.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.829+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded819ac40 con 0x7fded80ff780 2024-05-06T14:29:37.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.830+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded81072c0 con 0x7fded80ff780 2024-05-06T14:29:37.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.830+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fdec80046c0 con 0x7fded80ff780 2024-05-06T14:29:37.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.831+0000 7fdece7fc640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fdec406c6b0 0x7fdec406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:37.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.831+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdec8014070 con 0x7fded80ff780 2024-05-06T14:29:37.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.831+0000 7fdedd5a7640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fdec406c6b0 0x7fdec406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:37.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.831+0000 7fdedd5a7640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fdec406c6b0 0x7fdec406eb60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fded0000f80 tx=0x7fded0004a80 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:37.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:37.835+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdec8056570 con 0x7fded80ff780 2024-05-06T14:29:38.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.044+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fded819aec0 con 0x7fded80ff780 2024-05-06T14:29:38.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.044+0000 7fdece7fc640 1 -- 172.21.15.153:0/4021000931 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4760 (secure 0 0 0) 0x7fdec8059b90 con 0x7fded80ff780 2024-05-06T14:29:38.031 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:29:38.031 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14506,"name":"cephfs.smithi153.hiwroq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/4261352408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4261352408},{"type":"v1","addr":"172.21.15.153:6827","nonce":4261352408}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24287,"name":"cephfs.smithi190.quxdec","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.190:6827/804379542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":804379542},{"type":"v1","addr":"172.21.15.190:6827","nonce":804379542}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-06T14:29:10.325593+0000","modified":"2024-05-06T14:29:30.035749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14494,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14494":{"gid":14494,"name":"cephfs.smithi153.ftbnnk","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/179559541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":179559541},{"type":"v1","addr":"172.21.15.153:6829","nonce":179559541}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi190.psnzbz","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.190:6825/3762223844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3762223844},{"type":"v1","addr":"172.21.15.190:6825","nonce":3762223844}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-05-06T14:29:38.032 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 16 2024-05-06T14:29:38.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fdec406c6b0 msgr2=0x7fdec406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:38.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fdec406c6b0 0x7fdec406eb60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fded0000f80 tx=0x7fded0004a80 comp rx=0 tx=0).stop 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 msgr2=0x7fded8195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 0x7fded8195490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdec800e970 tx=0x7fdec800ed70 comp rx=0 tx=0).stop 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 shutdown_connections 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fdec406c6b0 0x7fdec406eb60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fded80ff780 0x7fded8195490 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 --2- 172.21.15.153:0/4021000931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fded80fe590 0x7fded8194f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.046+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 >> 172.21.15.153:0/4021000931 conn(0x7fded80f9b30 msgr2=0x7fded8068390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.047+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 shutdown_connections 2024-05-06T14:29:38.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:38.047+0000 7fdedf80b640 1 -- 172.21.15.153:0/4021000931 wait complete. 2024-05-06T14:29:38.046 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 2, 'flags': 18} 2024-05-06T14:29:38.046 INFO:teuthology.run_tasks:Running task kclient... 2024-05-06T14:29:38.143 INFO:tasks.kclient:Mounting kernel clients... 2024-05-06T14:29:38.143 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-05-06T14:29:38.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:38.143 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-05-06T14:29:38.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:38.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-05-06T14:29:38.224 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-05-06T14:29:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:29:38.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:38.226 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2024-05-06T14:29:38.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:38.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2024-05-06T14:29:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:38 smithi153 ceph-mon[27177]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-06T14:29:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:38 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:38 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:38 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1344637580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:29:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:38 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3691994962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-06T14:29:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:38 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2855295094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-06T14:29:38.310 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:38 smithi190 ceph-mon[33671]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-06T14:29:38.310 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:38 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:38.310 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:38 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:38.311 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:38 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1344637580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:29:38.311 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:38 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3691994962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-06T14:29:38.311 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:38 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2855295094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-06T14:29:38.318 INFO:teuthology.orchestra.run.smithi190.stderr:Cannot find device "ceph-brx" 2024-05-06T14:29:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:29:38.320 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-05-06T14:29:38.320 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-06T14:29:38.320 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:38.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-06T14:29:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:39.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:39 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/4021000931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-06T14:29:39.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:39 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/4021000931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-06T14:29:40.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:40 smithi153 ceph-mon[27177]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-06T14:29:40.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:40.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:40 smithi190 ceph-mon[33671]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-06T14:29:40.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.211+0000 7f243df2d700 1 -- 172.21.15.153:0/2333796545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24381003d0 msgr2=0x7f24381007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.211+0000 7f243df2d700 1 --2- 172.21.15.153:0/2333796545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24381003d0 0x7f24381007a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f242c009910 tx=0x7f242c009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.211+0000 7f243df2d700 1 -- 172.21.15.153:0/2333796545 shutdown_connections 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.211+0000 7f243df2d700 1 --2- 172.21.15.153:0/2333796545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2438100d70 0x7f2438104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.211+0000 7f243df2d700 1 --2- 172.21.15.153:0/2333796545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24381003d0 0x7f24381007a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.211+0000 7f243df2d700 1 -- 172.21.15.153:0/2333796545 >> 172.21.15.153:0/2333796545 conn(0x7f24380fbc90 msgr2=0x7f24380fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 -- 172.21.15.153:0/2333796545 shutdown_connections 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 -- 172.21.15.153:0/2333796545 wait complete. 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 Processor -- start 2024-05-06T14:29:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 -- start start 2024-05-06T14:29:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24381003d0 0x7f2438197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 0x7f2438198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2438198ab0 con 0x7f2438100d70 2024-05-06T14:29:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.212+0000 7f243df2d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2438198bf0 con 0x7f24381003d0 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 0x7f2438198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 0x7f2438198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48112/0 (socket says 172.21.15.153:48112) 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 -- 172.21.15.153:0/3745009301 learned_addr learned my addr 172.21.15.153:0/3745009301 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f24377fe700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24381003d0 0x7f2438197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 -- 172.21.15.153:0/3745009301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24381003d0 msgr2=0x7f2438197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24381003d0 0x7f2438197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 -- 172.21.15.153:0/3745009301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0095f0 con 0x7f2438100d70 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2436ffd700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 0x7f2438198460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f242800c960 tx=0x7f242800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2428007a70 con 0x7f2438100d70 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2428007bd0 con 0x7f2438100d70 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243819ca40 con 0x7f2438100d70 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2428018630 con 0x7f2438100d70 2024-05-06T14:29:41.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.213+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243819cf90 con 0x7f2438100d70 2024-05-06T14:29:41.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.214+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24381044c0 con 0x7f2438100d70 2024-05-06T14:29:41.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.215+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f242800fa90 con 0x7f2438100d70 2024-05-06T14:29:41.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.215+0000 7f2434ff9700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f242006c680 0x7f242006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:41.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.215+0000 7f24377fe700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f242006c680 0x7f242006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:41.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.215+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f242808b920 con 0x7f2438100d70 2024-05-06T14:29:41.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.216+0000 7f24377fe700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f242006c680 0x7f242006eb30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f242c00b5c0 tx=0x7f242c01a040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:41.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.220+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2428055fa0 con 0x7f2438100d70 2024-05-06T14:29:41.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.454+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f243804ea50 con 0x7f2438100d70 2024-05-06T14:29:41.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.455+0000 7f2434ff9700 1 -- 172.21.15.153:0/3745009301 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f24280595c0 con 0x7f2438100d70 2024-05-06T14:29:41.442 INFO:teuthology.orchestra.run.smithi153.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f242006c680 msgr2=0x7f242006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f242006c680 0x7f242006eb30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f242c00b5c0 tx=0x7f242c01a040 comp rx=0 tx=0).stop 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 msgr2=0x7f2438198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 0x7f2438198460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f242800c960 tx=0x7f242800cd20 comp rx=0 tx=0).stop 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 shutdown_connections 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f242006c680 0x7f242006eb30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24381003d0 0x7f2438197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 --2- 172.21.15.153:0/3745009301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2438100d70 0x7f2438198460 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 >> 172.21.15.153:0/3745009301 conn(0x7f24380fbc90 msgr2=0x7f24380fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 shutdown_connections 2024-05-06T14:29:41.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:41.457+0000 7f243df2d700 1 -- 172.21.15.153:0/3745009301 wait complete. 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:42.085 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:42 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3745009301' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-06T14:29:42.106 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-06T14:29:42.106 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-06T14:29:42.106 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi153.front.sepia.ceph.com 2024-05-06T14:29:42.106 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-06T14:29:42.106 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:42.107 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-06T14:29:42.107 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-06T14:29:42.107 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-06T14:29:42.107 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-06T14:29:42.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:42.107 DEBUG:teuthology.orchestra.run.smithi153:> ip addr 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b2 brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:42.123 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f0 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b3 brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f1 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:ea brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:eb brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:42.124 INFO:teuthology.orchestra.run.smithi153.stdout: inet 172.21.15.153/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-06T14:29:42.125 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft 42149sec preferred_lft 42149sec 2024-05-06T14:29:42.125 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::c681:6dfa:26e2:6476/64 scope link noprefixroute 2024-05-06T14:29:42.125 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-05-06T14:29:42.126 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add name ceph-brx type bridge 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr flush dev ceph-brx 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set ceph-brx up 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-06T14:29:42.126 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-05-06T14:29:42.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:42.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:42.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:42.307 DEBUG:teuthology.orchestra.run.smithi153:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-06T14:29:42.332 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-05-06T14:29:42.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:42.334 DEBUG:teuthology.orchestra.run.smithi153:> route 2024-05-06T14:29:42.411 INFO:teuthology.orchestra.run.smithi153.stdout:Kernel IP routing table 2024-05-06T14:29:42.411 INFO:teuthology.orchestra.run.smithi153.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-06T14:29:42.411 INFO:teuthology.orchestra.run.smithi153.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-06T14:29:42.411 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-06T14:29:42.411 INFO:teuthology.orchestra.run.smithi153.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-06T14:29:42.412 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:42.412 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-05-06T14:29:42.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-06T14:29:42.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-06T14:29:42.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-06T14:29:42.412 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:29:42.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:42 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3745009301' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-06T14:29:42.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:42.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:42.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:42.685 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-05-06T14:29:42.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:42.701 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list-id 2024-05-06T14:29:42.757 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:42.757 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-05-06T14:29:42.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-06T14:29:42.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-06T14:29:42.757 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-05-06T14:29:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:42.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:42.872 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-06T14:29:42.873 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-05-06T14:29:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:42.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:42.995 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:42.995 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-05-06T14:29:42.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set brx.0 up 2024-05-06T14:29:42.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set dev brx.0 master ceph-brx 2024-05-06T14:29:42.995 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-05-06T14:29:43.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:43.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:43.070 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-06T14:29:43.070 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-06T14:29:43.086 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-06T14:29:43.086 DEBUG:teuthology.orchestra.run.smithi153:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:43.145 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-06T14:29:43.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:43.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-06T14:29:43.237 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-06T14:29:43.237 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-06T14:29:43.237 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-05-06T14:29:43.337 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-06T14:29:43.337 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-06T14:29:43.337 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-05-06T14:29:43.338 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.190:3300" 2024-05-06T14:29:43.338 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-06T14:29:43.342 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-06T14:29:43.342 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-06T14:29:43.343 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-05-06T14:29:43.343 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.190:3300" 2024-05-06T14:29:43.343 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-06T14:29:43.343 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.153:6789,172.21.15.190:6789" 2024-05-06T14:29:43.343 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.153:6789,172.21.15.190:6789:/ 2024-05-06T14:29:43.343 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=eacfcfb6-0bb3-11ef-bc97-c7b262605968" will pass to kernel 2024-05-06T14:29:43.344 INFO:teuthology.orchestra.run.smithi153.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-06T14:29:43.344 INFO:teuthology.orchestra.run.smithi153.stdout: You just mounted a file system that supports labels which does not 2024-05-06T14:29:43.344 INFO:teuthology.orchestra.run.smithi153.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-06T14:29:43.344 INFO:teuthology.orchestra.run.smithi153.stdout: applications will generate AVC messages and not be allowed access to 2024-05-06T14:29:43.344 INFO:teuthology.orchestra.run.smithi153.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-06T14:29:43.345 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-06T14:29:43.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:43.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:43.373 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-06T14:29:43.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-06T14:29:43.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:43.474 INFO:teuthology.orchestra.run.smithi153.stdout:client24323 2024-05-06T14:29:43.475 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:29:43.475 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-06T14:29:43.492 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-06T14:29:43.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-06T14:29:43.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:43.593 INFO:teuthology.orchestra.run.smithi153.stdout:client24323 2024-05-06T14:29:43.593 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-06T14:29:43.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-06T14:29:43.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:43.653 INFO:teuthology.orchestra.run.smithi153.stdout:client24323 2024-05-06T14:29:43.654 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968.client24323/status') 2024-05-06T14:29:43.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:43.714 INFO:teuthology.orchestra.run.smithi153.stdout:instance: client.24323 (3)172.21.15.153:0/848054582 2024-05-06T14:29:43.714 INFO:teuthology.orchestra.run.smithi153.stdout:blocklisted: false 2024-05-06T14:29:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records in 2024-05-06T14:29:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records out 2024-05-06T14:29:43.714 INFO:teuthology.orchestra.run.smithi153.stderr:71 bytes copied, 2.2584e-05 s, 3.1 MB/s 2024-05-06T14:29:43.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:43.721 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-05-06T14:29:43.721 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-06T14:29:43.721 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:43.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-06T14:29:43.927 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:44.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:44 smithi153 ceph-mon[27177]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-06T14:29:44.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:44 smithi190 ceph-mon[33671]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-06T14:29:45.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 -- 172.21.15.153:0/626072109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 msgr2=0x7fc644104d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:45.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 --2- 172.21.15.153:0/626072109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc644104d50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc638009910 tx=0x7fc638009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 -- 172.21.15.153:0/626072109 shutdown_connections 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 --2- 172.21.15.153:0/626072109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc644104d50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 --2- 172.21.15.153:0/626072109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6441003e0 0x7fc6441007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 -- 172.21.15.153:0/626072109 >> 172.21.15.153:0/626072109 conn(0x7fc6440fbca0 msgr2=0x7fc6440fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 -- 172.21.15.153:0/626072109 shutdown_connections 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.793+0000 7fc648bde700 1 -- 172.21.15.153:0/626072109 wait complete. 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc648bde700 1 Processor -- start 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc648bde700 1 -- start start 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc648bde700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6441003e0 0x7fc644197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc648bde700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc6441984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc648bde700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc644198b90 con 0x7fc644100d80 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc648bde700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64419c920 con 0x7fc6441003e0 2024-05-06T14:29:45.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc6441984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc6441984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48136/0 (socket says 172.21.15.153:48136) 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 -- 172.21.15.153:0/2420339309 learned_addr learned my addr 172.21.15.153:0/2420339309 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc64259c700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6441003e0 0x7fc644197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 -- 172.21.15.153:0/2420339309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6441003e0 msgr2=0x7fc644197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6441003e0 0x7fc644197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 -- 172.21.15.153:0/2420339309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380095f0 con 0x7fc644100d80 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.794+0000 7fc641d9b700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc6441984b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc6380092e0 tx=0x7fc63800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.795+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc63801d070 con 0x7fc644100d80 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.795+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64419cba0 con 0x7fc644100d80 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.795+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc638005dc0 con 0x7fc644100d80 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.795+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc638017400 con 0x7fc644100d80 2024-05-06T14:29:45.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.795+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64419d090 con 0x7fc644100d80 2024-05-06T14:29:45.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.796+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6441044d0 con 0x7fc644100d80 2024-05-06T14:29:45.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.796+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fc638003a60 con 0x7fc644100d80 2024-05-06T14:29:45.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.796+0000 7fc6337fe700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc62c06c680 0x7fc62c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:45.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.797+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc63808ccd0 con 0x7fc644100d80 2024-05-06T14:29:45.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.797+0000 7fc64259c700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc62c06c680 0x7fc62c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:45.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.797+0000 7fc64259c700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc62c06c680 0x7fc62c06eb30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc634005950 tx=0x7fc63400b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:45.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:45.801+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc638057350 con 0x7fc644100d80 2024-05-06T14:29:46.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.035+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc64404ea50 con 0x7fc644100d80 2024-05-06T14:29:46.023 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:46 smithi153 ceph-mon[27177]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-05-06T14:29:46.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.036+0000 7fc6337fe700 1 -- 172.21.15.153:0/2420339309 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fc63805a970 con 0x7fc644100d80 2024-05-06T14:29:46.023 INFO:teuthology.orchestra.run.smithi153.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc62c06c680 msgr2=0x7fc62c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc62c06c680 0x7fc62c06eb30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc634005950 tx=0x7fc63400b410 comp rx=0 tx=0).stop 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 msgr2=0x7fc6441984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc6441984b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc6380092e0 tx=0x7fc63800bd70 comp rx=0 tx=0).stop 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 shutdown_connections 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fc62c06c680 0x7fc62c06eb30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6441003e0 0x7fc644197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 --2- 172.21.15.153:0/2420339309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc644100d80 0x7fc6441984b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 >> 172.21.15.153:0/2420339309 conn(0x7fc6440fbca0 msgr2=0x7fc6440fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 shutdown_connections 2024-05-06T14:29:46.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:46.039+0000 7fc648bde700 1 -- 172.21.15.153:0/2420339309 wait complete. 2024-05-06T14:29:46.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:46 smithi190 ceph-mon[33671]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi190.front.sepia.ceph.com 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-06T14:29:46.681 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-06T14:29:46.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:46.682 DEBUG:teuthology.orchestra.run.smithi190:> ip addr 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: inet 127.0.0.1/8 scope host lo 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 ::1/128 scope host 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d0 brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f0 2024-05-06T14:29:46.700 INFO:teuthology.orchestra.run.smithi190.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:d9:36:d1 brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: altname enp7s0f1 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:62 brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: link/ether 0c:c4:7a:88:77:63 brd ff:ff:ff:ff:ff:ff 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: inet 172.21.15.190/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft 42124sec preferred_lft 42124sec 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: inet6 fe80::b658:4a7:d9fd:b6fc/64 scope link noprefixroute 2024-05-06T14:29:46.701 INFO:teuthology.orchestra.run.smithi190.stdout: valid_lft forever preferred_lft forever 2024-05-06T14:29:46.701 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add name ceph-brx type bridge 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip addr flush dev ceph-brx 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set ceph-brx up 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-06T14:29:46.702 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-05-06T14:29:46.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:46.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:46.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:46.887 DEBUG:teuthology.orchestra.run.smithi190:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-06T14:29:46.912 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-05-06T14:29:46.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:46.913 DEBUG:teuthology.orchestra.run.smithi190:> route 2024-05-06T14:29:46.977 INFO:teuthology.orchestra.run.smithi190.stdout:Kernel IP routing table 2024-05-06T14:29:46.977 INFO:teuthology.orchestra.run.smithi190.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-06T14:29:46.977 INFO:teuthology.orchestra.run.smithi190.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-06T14:29:46.977 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-06T14:29:46.977 INFO:teuthology.orchestra.run.smithi190.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-06T14:29:46.978 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:46.978 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-05-06T14:29:46.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-06T14:29:46.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-06T14:29:46.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-06T14:29:46.978 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-05-06T14:29:47.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:47.072 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:47 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2420339309' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-06T14:29:47.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:47 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2420339309' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-06T14:29:47.318 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:47.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:47.325 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2024-05-06T14:29:47.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:47.340 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list-id 2024-05-06T14:29:47.395 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:47.395 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-05-06T14:29:47.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-06T14:29:47.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-06T14:29:47.395 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-05-06T14:29:47.484 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:47.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:47.512 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-06T14:29:47.513 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-05-06T14:29:47.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:47.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:47.636 DEBUG:teuthology.orchestra.run.smithi190:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-06T14:29:47.636 DEBUG:teuthology.orchestra.run.smithi190:> set -e 2024-05-06T14:29:47.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set brx.0 up 2024-05-06T14:29:47.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link set dev brx.0 master ceph-brx 2024-05-06T14:29:47.637 DEBUG:teuthology.orchestra.run.smithi190:> ') 2024-05-06T14:29:47.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:47.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:47.711 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-06T14:29:47.711 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:47.727 INFO:teuthology.orchestra.run.smithi190.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-06T14:29:47.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-06T14:29:47.727 DEBUG:teuthology.orchestra.run.smithi190:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:47.782 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-06T14:29:47.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:47.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-06T14:29:47.881 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-06T14:29:47.882 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-06T14:29:47.882 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2024-05-06T14:29:47.985 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-06T14:29:47.985 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-06T14:29:47.985 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2024-05-06T14:29:47.985 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.190:3300" 2024-05-06T14:29:47.986 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-06T14:29:47.990 INFO:teuthology.orchestra.run.smithi190.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-05-06T14:29:47.990 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-05-06T14:29:47.990 INFO:teuthology.orchestra.run.smithi190.stdout:invalid new device string format 2024-05-06T14:29:47.990 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.190:3300" 2024-05-06T14:29:47.990 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-06T14:29:47.991 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: resolved to: "172.21.15.153:6789,172.21.15.190:6789" 2024-05-06T14:29:47.991 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.153:6789,172.21.15.190:6789:/ 2024-05-06T14:29:47.991 INFO:teuthology.orchestra.run.smithi190.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=eacfcfb6-0bb3-11ef-bc97-c7b262605968" will pass to kernel 2024-05-06T14:29:47.995 INFO:teuthology.orchestra.run.smithi190.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-06T14:29:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: You just mounted a file system that supports labels which does not 2024-05-06T14:29:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-06T14:29:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: applications will generate AVC messages and not be allowed access to 2024-05-06T14:29:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-06T14:29:47.996 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-06T14:29:47.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:29:47.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:48.020 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-06T14:29:48.108 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-06T14:29:48.110 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:48.116 INFO:teuthology.orchestra.run.smithi190.stdout:client24329 2024-05-06T14:29:48.116 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:29:48.117 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-06T14:29:48.132 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-06T14:29:48.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:48 smithi190 ceph-mon[33671]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-06T14:29:48.220 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-06T14:29:48.222 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:48.228 INFO:teuthology.orchestra.run.smithi190.stdout:client24329 2024-05-06T14:29:48.229 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-06T14:29:48.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:48 smithi153 ceph-mon[27177]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-06T14:29:48.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-06T14:29:48.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:48.324 INFO:teuthology.orchestra.run.smithi190.stdout:client24329 2024-05-06T14:29:48.324 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968.client24329/status') 2024-05-06T14:29:48.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-06T14:29:48.381 INFO:teuthology.orchestra.run.smithi190.stdout:instance: client.24329 (3)192.168.0.1:0/542645738 2024-05-06T14:29:48.381 INFO:teuthology.orchestra.run.smithi190.stdout:blocklisted: false 2024-05-06T14:29:48.382 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2024-05-06T14:29:48.382 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2024-05-06T14:29:48.382 INFO:teuthology.orchestra.run.smithi190.stderr:69 bytes copied, 2.2268e-05 s, 3.1 MB/s 2024-05-06T14:29:48.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-05-06T14:29:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-06T14:29:48.389 INFO:teuthology.run_tasks:Running task print... 2024-05-06T14:29:48.406 INFO:teuthology.task.print:**** done client 2024-05-06T14:29:48.406 INFO:teuthology.run_tasks:Running task parallel... 2024-05-06T14:29:48.416 INFO:teuthology.task.parallel:starting parallel... 2024-05-06T14:29:48.416 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-05-06T14:29:48.417 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-05-06T14:29:48.417 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:29:48.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-05-06T14:29:48.418 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-05-06T14:29:48.418 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-05-06T14:29:48.422 INFO:tasks.workunit:Pulling workunits from ref 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:29:48.422 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-06T14:29:48.422 INFO:tasks.workunit:timeout=3h 2024-05-06T14:29:48.422 INFO:tasks.workunit:cleanup=True 2024-05-06T14:29:48.423 DEBUG:teuthology.orchestra.run.smithi153:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 7dh/125d Inode: 1 Links: 2 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-05-06 14:29:15.353970887 +0000 2024-05-06T14:29:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-05-06 14:29:43.381537300 +0000 2024-05-06T14:29:48.445 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-05-06 14:29:15.353970887 +0000 2024-05-06T14:29:48.445 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-06T14:29:48.445 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-06T14:29:48.513 DEBUG:teuthology.orchestra.run.smithi190:> stat -- /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:48.530 INFO:teuthology.orchestra.run.smithi190.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:48.530 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-05-06T14:29:48.530 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-05-06T14:29:48.530 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T14:29:48.530 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-06T14:29:48.530 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-06T14:29:48.531 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-05-06 14:29:48.521440457 +0000 2024-05-06T14:29:48.531 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-05-06 14:29:48.521440457 +0000 2024-05-06T14:29:48.531 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-05-06 14:29:15.353970887 +0000 2024-05-06T14:29:48.531 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-05-06T14:29:48.531 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-05-06T14:29:48.598 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:29:48.599 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:29:48.614 INFO:tasks.workunit.client.0.smithi153.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-06T14:29:48.645 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:48.654 INFO:tasks.workunit.client.1.smithi190.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-05-06T14:29:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 -- 172.21.15.153:0/2699875217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428103500 msgr2=0x7f94281058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 --2- 172.21.15.153:0/2699875217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428103500 0x7f94281058e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9410009910 tx=0x7f9410009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 -- 172.21.15.153:0/2699875217 shutdown_connections 2024-05-06T14:29:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 --2- 172.21.15.153:0/2699875217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428103500 0x7f94281058e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 --2- 172.21.15.153:0/2699875217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428100be0 0x7f9428102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 -- 172.21.15.153:0/2699875217 >> 172.21.15.153:0/2699875217 conn(0x7f94280fa600 msgr2=0x7f94280fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 -- 172.21.15.153:0/2699875217 shutdown_connections 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.106+0000 7f942ceba700 1 -- 172.21.15.153:0/2699875217 wait complete. 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942ceba700 1 Processor -- start 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942ceba700 1 -- start start 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942ceba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 0x7f9428197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942ceba700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428103500 0x7f94281980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942ceba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94281986e0 con 0x7f9428100be0 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942ceba700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9428198820 con 0x7f9428103500 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 0x7f9428197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 0x7f9428197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48050/0 (socket says 172.21.15.153:48050) 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942659c700 1 -- 172.21.15.153:0/4066454460 learned_addr learned my addr 172.21.15.153:0/4066454460 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f941ffff700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428103500 0x7f94281980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942659c700 1 -- 172.21.15.153:0/4066454460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428103500 msgr2=0x7f94281980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942659c700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428103500 0x7f94281980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.107+0000 7f942659c700 1 -- 172.21.15.153:0/4066454460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94100095f0 con 0x7f9428100be0 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f941ffff700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428103500 0x7f94281980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f942659c700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 0x7f9428197b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f941800b4b0 tx=0x7f941800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9418011800 con 0x7f9428100be0 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9418011e10 con 0x7f9428100be0 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942819d2d0 con 0x7f9428100be0 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941800f2e0 con 0x7f9428100be0 2024-05-06T14:29:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.108+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942819d8a0 con 0x7f9428100be0 2024-05-06T14:29:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.109+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94280fc1c0 con 0x7f9428100be0 2024-05-06T14:29:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.109+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f9418010380 con 0x7f9428100be0 2024-05-06T14:29:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.110+0000 7f941f7fe700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f941406c680 0x7f941406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.110+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f941808b550 con 0x7f9428100be0 2024-05-06T14:29:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.110+0000 7f941ffff700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f941406c680 0x7f941406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.110+0000 7f941ffff700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f941406c680 0x7f941406eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f941000b5c0 tx=0x7f9410005e20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.114+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9418059660 con 0x7f9428100be0 2024-05-06T14:29:50.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:50 smithi153 ceph-mon[27177]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-05-06T14:29:50.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.304+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f942802cc30 con 0x7f9428100be0 2024-05-06T14:29:50.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.304+0000 7f941f7fe700 1 -- 172.21.15.153:0/4066454460 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f94180591f0 con 0x7f9428100be0 2024-05-06T14:29:50.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f941406c680 msgr2=0x7f941406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:50.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f941406c680 0x7f941406eb30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f941000b5c0 tx=0x7f9410005e20 comp rx=0 tx=0).stop 2024-05-06T14:29:50.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 msgr2=0x7f9428197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:50.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 0x7f9428197b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f941800b4b0 tx=0x7f941800b870 comp rx=0 tx=0).stop 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 shutdown_connections 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9428100be0 0x7f9428197b80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f941406c680 0x7f941406eb30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 --2- 172.21.15.153:0/4066454460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9428103500 0x7f94281980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 >> 172.21.15.153:0/4066454460 conn(0x7f94280fa600 msgr2=0x7f94280ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 shutdown_connections 2024-05-06T14:29:50.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:50.306+0000 7f942ceba700 1 -- 172.21.15.153:0/4066454460 wait complete. 2024-05-06T14:29:50.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:50 smithi190 ceph-mon[33671]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-05-06T14:29:50.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-05-06T14:29:51.091 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:52.187 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:52 smithi153 ceph-mon[27177]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-05-06T14:29:52.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:52 smithi190 ceph-mon[33671]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.691+0000 7fd6a18f1700 1 -- 172.21.15.153:0/4244550571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c100290 msgr2=0x7fd69c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.691+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/4244550571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c100290 0x7fd69c1006e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd684009910 tx=0x7fd684009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.691+0000 7fd6a18f1700 1 -- 172.21.15.153:0/4244550571 shutdown_connections 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.691+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/4244550571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c100290 0x7fd69c1006e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.691+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/4244550571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c0ff090 0x7fd69c0ff4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 -- 172.21.15.153:0/4244550571 >> 172.21.15.153:0/4244550571 conn(0x7fd69c0fa660 msgr2=0x7fd69c0fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 -- 172.21.15.153:0/4244550571 shutdown_connections 2024-05-06T14:29:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 -- 172.21.15.153:0/4244550571 wait complete. 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 Processor -- start 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 -- start start 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 0x7fd69c195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100290 0x7fd69c195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1964e0 con 0x7fd69c0ff090 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd6a18f1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c196620 con 0x7fd69c100290 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd69affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 0x7fd69c195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd69affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 0x7fd69c195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48064/0 (socket says 172.21.15.153:48064) 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.692+0000 7fd69affd700 1 -- 172.21.15.153:0/1564924340 learned_addr learned my addr 172.21.15.153:0/1564924340 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd693fff700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100290 0x7fd69c195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd69affd700 1 -- 172.21.15.153:0/1564924340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100290 msgr2=0x7fd69c195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd69affd700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100290 0x7fd69c195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd69affd700 1 -- 172.21.15.153:0/1564924340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6840095f0 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd69affd700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 0x7fd69c195980 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00d900 tx=0x7fd68c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68c0041f0 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd68c004350 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68c003d20 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c19b0d0 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.693+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c19b620 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.694+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c107be0 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.695+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd68c009730 con 0x7fd69c0ff090 2024-05-06T14:29:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.695+0000 7fd6a08ef700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd68806c680 0x7fd68806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.695+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd68c08b6f0 con 0x7fd69c0ff090 2024-05-06T14:29:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.695+0000 7fd693fff700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd68806c680 0x7fd68806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.695+0000 7fd693fff700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd68806c680 0x7fd68806eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd684006010 tx=0x7fd68400b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.699+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd68c059880 con 0x7fd69c0ff090 2024-05-06T14:29:52.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.887+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fd69c19bd10 con 0x7fd69c0ff090 2024-05-06T14:29:52.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.888+0000 7fd6a08ef700 1 -- 172.21.15.153:0/1564924340 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd68c059410 con 0x7fd69c0ff090 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd68806c680 msgr2=0x7fd68806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd68806c680 0x7fd68806eb30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd684006010 tx=0x7fd68400b540 comp rx=0 tx=0).stop 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 msgr2=0x7fd69c195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 0x7fd69c195980 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00d900 tx=0x7fd68c00dc10 comp rx=0 tx=0).stop 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 shutdown_connections 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd69c0ff090 0x7fd69c195980 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd68806c680 0x7fd68806eb30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 --2- 172.21.15.153:0/1564924340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd69c100290 0x7fd69c195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 >> 172.21.15.153:0/1564924340 conn(0x7fd69c0fa660 msgr2=0x7fd69c068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:52.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 shutdown_connections 2024-05-06T14:29:52.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:52.890+0000 7fd6a18f1700 1 -- 172.21.15.153:0/1564924340 wait complete. 2024-05-06T14:29:53.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-05-06T14:29:53.782 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:54.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:54 smithi153 ceph-mon[27177]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-06T14:29:54.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:54 smithi190 ceph-mon[33671]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-06T14:29:55.171 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:55.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 -- 172.21.15.153:0/4002339690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 msgr2=0x7f400c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 --2- 172.21.15.153:0/4002339690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c0ff460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009910 tx=0x7f3ffc009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 -- 172.21.15.153:0/4002339690 shutdown_connections 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 --2- 172.21.15.153:0/4002339690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f400c100250 0x7f400c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 --2- 172.21.15.153:0/4002339690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c0ff460 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 -- 172.21.15.153:0/4002339690 >> 172.21.15.153:0/4002339690 conn(0x7f400c0fa5c0 msgr2=0x7f400c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 -- 172.21.15.153:0/4002339690 shutdown_connections 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.374+0000 7f4011f61700 1 -- 172.21.15.153:0/4002339690 wait complete. 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f4011f61700 1 Processor -- start 2024-05-06T14:29:55.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f4011f61700 1 -- start start 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f4011f61700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f4011f61700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f400c100250 0x7f400c101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f4011f61700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c101b70 con 0x7f400c0ff050 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f4011f61700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c101cb0 con 0x7f400c100250 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48078/0 (socket says 172.21.15.153:48078) 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 -- 172.21.15.153:0/3359179525 learned_addr learned my addr 172.21.15.153:0/3359179525 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 -- 172.21.15.153:0/3359179525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f400c100250 msgr2=0x7f400c101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f400c100250 0x7f400c101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 -- 172.21.15.153:0/3359179525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc0095f0 con 0x7f400c0ff050 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.375+0000 7f400b7fe700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c102f90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00bda0 tx=0x7f3ffc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:55.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.376+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ffc01d070 con 0x7f400c0ff050 2024-05-06T14:29:55.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.376+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ffc004c50 con 0x7f400c0ff050 2024-05-06T14:29:55.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.376+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c101f30 con 0x7f400c0ff050 2024-05-06T14:29:55.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.376+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ffc00f690 con 0x7f400c0ff050 2024-05-06T14:29:55.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.376+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c1024a0 con 0x7f400c0ff050 2024-05-06T14:29:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.377+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c107c20 con 0x7f400c0ff050 2024-05-06T14:29:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.377+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f3ffc00f900 con 0x7f400c0ff050 2024-05-06T14:29:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.377+0000 7f4008ff9700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f3ff406c680 0x7f3ff406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.378+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3ffc08cfa0 con 0x7f400c0ff050 2024-05-06T14:29:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.378+0000 7f400affd700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f3ff406c680 0x7f3ff406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.378+0000 7f400affd700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f3ff406c680 0x7f3ff406eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40000097b0 tx=0x7f4000006d20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:55.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.382+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3ffc05b0b0 con 0x7f400c0ff050 2024-05-06T14:29:55.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:29:55.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.572+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f400c107e10 con 0x7f400c0ff050 2024-05-06T14:29:55.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.573+0000 7f4008ff9700 1 -- 172.21.15.153:0/3359179525 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f3ffc05ac40 con 0x7f400c0ff050 2024-05-06T14:29:55.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f3ff406c680 msgr2=0x7f3ff406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:55.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f3ff406c680 0x7f3ff406eb30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40000097b0 tx=0x7f4000006d20 comp rx=0 tx=0).stop 2024-05-06T14:29:55.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 msgr2=0x7f400c102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:55.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c102f90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00bda0 tx=0x7f3ffc00be80 comp rx=0 tx=0).stop 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 shutdown_connections 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c0ff050 0x7f400c102f90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f3ff406c680 0x7f3ff406eb30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 --2- 172.21.15.153:0/3359179525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f400c100250 0x7f400c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 >> 172.21.15.153:0/3359179525 conn(0x7f400c0fa5c0 msgr2=0x7f400c068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 shutdown_connections 2024-05-06T14:29:55.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:55.575+0000 7f4011f61700 1 -- 172.21.15.153:0/3359179525 wait complete. 2024-05-06T14:29:56.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:56 smithi153 ceph-mon[27177]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-06T14:29:56.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-05-06T14:29:56.389 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:29:56.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:56 smithi190 ceph-mon[33671]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-06T14:29:57.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 -- 172.21.15.153:0/1696030457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8101370 msgr2=0x7fbcd81017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:57.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 --2- 172.21.15.153:0/1696030457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8101370 0x7fbcd81017c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009910 tx=0x7fbcc0009c20 comp rx=0 tx=0).stop 2024-05-06T14:29:57.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 -- 172.21.15.153:0/1696030457 shutdown_connections 2024-05-06T14:29:57.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 --2- 172.21.15.153:0/1696030457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8101370 0x7fbcd81017c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:57.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 --2- 172.21.15.153:0/1696030457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd8100170 0x7fbcd8100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 -- 172.21.15.153:0/1696030457 >> 172.21.15.153:0/1696030457 conn(0x7fbcd80fb6e0 msgr2=0x7fbcd80fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 -- 172.21.15.153:0/1696030457 shutdown_connections 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.894+0000 7fbcddf04700 1 -- 172.21.15.153:0/1696030457 wait complete. 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcddf04700 1 Processor -- start 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcddf04700 1 -- start start 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcddf04700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 0x7fbcd8074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcddf04700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd8101370 0x7fbcd8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcddf04700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8073680 con 0x7fbcd8100170 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcddf04700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd80737c0 con 0x7fbcd8101370 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcd77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 0x7fbcd8074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcd77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 0x7fbcd8074af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48108/0 (socket says 172.21.15.153:48108) 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.895+0000 7fbcd77fe700 1 -- 172.21.15.153:0/165902536 learned_addr learned my addr 172.21.15.153:0/165902536 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd77fe700 1 -- 172.21.15.153:0/165902536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd8101370 msgr2=0x7fbcd8073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:29:57.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd77fe700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd8101370 0x7fbcd8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd77fe700 1 -- 172.21.15.153:0/165902536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00095f0 con 0x7fbcd8100170 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd77fe700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 0x7fbcd8074af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800b480 tx=0x7fbcc800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc80118c0 con 0x7fbcd8100170 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbcc8010380 con 0x7fbcd8100170 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc800f4f0 con 0x7fbcd8100170 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8073aa0 con 0x7fbcd8100170 2024-05-06T14:29:57.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.896+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd819da90 con 0x7fbcd8100170 2024-05-06T14:29:57.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.897+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd8108d60 con 0x7fbcd8100170 2024-05-06T14:29:57.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.897+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fbcc80104f0 con 0x7fbcd8100170 2024-05-06T14:29:57.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.898+0000 7fbcd57fa700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbcc406c680 0x7fbcc406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:29:57.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.898+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbcc808b470 con 0x7fbcd8100170 2024-05-06T14:29:57.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.898+0000 7fbcceffd700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbcc406c680 0x7fbcc406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:29:57.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.898+0000 7fbcceffd700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbcc406c680 0x7fbcc406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b5c0 tx=0x7fbcc0005960 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:29:57.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:57.902+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbcc8059600 con 0x7fbcd8100170 2024-05-06T14:29:58.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:29:58.113+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcd8105cc0 con 0x7fbcc406c680 2024-05-06T14:29:58.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:58 smithi190 ceph-mon[33671]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-06T14:29:58.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:58 smithi153 ceph-mon[27177]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s wr, 0 op/s 2024-05-06T14:29:59.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:29:59 smithi190 ceph-mon[33671]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:29:59.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:29:59 smithi153 ceph-mon[27177]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:00.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:00 smithi190 ceph-mon[33671]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-05-06T14:30:00.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:00 smithi190 ceph-mon[33671]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-05-06T14:30:00.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:00 smithi190 ceph-mon[33671]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-05-06T14:30:00.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:00 smithi190 ceph-mon[33671]: fs cephfs has deprecated feature inline_data enabled. 2024-05-06T14:30:00.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:00 smithi153 ceph-mon[27177]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-05-06T14:30:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:00 smithi153 ceph-mon[27177]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-05-06T14:30:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:00 smithi153 ceph-mon[27177]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-05-06T14:30:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:00 smithi153 ceph-mon[27177]: fs cephfs has deprecated feature inline_data enabled. 2024-05-06T14:30:02.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:02 smithi190 ceph-mon[33671]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-05-06T14:30:02.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:02 smithi153 ceph-mon[27177]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-05-06T14:30:04.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:04 smithi190 ceph-mon[33671]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-05-06T14:30:04.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:04 smithi153 ceph-mon[27177]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-05-06T14:30:06.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:06 smithi190 ceph-mon[33671]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:06.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:06 smithi153 ceph-mon[27177]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:08.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:08 smithi190 ceph-mon[33671]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:08.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:08 smithi153 ceph-mon[27177]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:10.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:10 smithi190 ceph-mon[33671]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:10.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:10.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:10 smithi153 ceph-mon[27177]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:10.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:12.377 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:12 smithi190 ceph-mon[33671]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:12.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:12 smithi153 ceph-mon[27177]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:14.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:14 smithi190 ceph-mon[33671]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:14.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:14 smithi153 ceph-mon[27177]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:16.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:16 smithi190 ceph-mon[33671]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:16.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:16 smithi153 ceph-mon[27177]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:18.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:18 smithi190 ceph-mon[33671]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:18 smithi153 ceph-mon[27177]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:20.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:20 smithi190 ceph-mon[33671]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:20.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:20 smithi153 ceph-mon[27177]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:22.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:22 smithi190 ceph-mon[33671]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:22.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:22 smithi153 ceph-mon[27177]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:24.423 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:24 smithi190 ceph-mon[33671]: pgmap v214: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-05-06T14:30:24.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:24 smithi153 ceph-mon[27177]: pgmap v214: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-05-06T14:30:25.432 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:25 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:25.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:25 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:26.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:26 smithi190 ceph-mon[33671]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-05-06T14:30:26.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:26 smithi153 ceph-mon[27177]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-05-06T14:30:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.573+0000 7fbcd57fa700 1 -- 172.21.15.153:0/165902536 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fbcd8105cc0 con 0x7fbcc406c680 2024-05-06T14:30:27.563 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbcc406c680 msgr2=0x7fbcc406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbcc406c680 0x7fbcc406eb30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b5c0 tx=0x7fbcc0005960 comp rx=0 tx=0).stop 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 msgr2=0x7fbcd8074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 0x7fbcd8074af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800b480 tx=0x7fbcc800b840 comp rx=0 tx=0).stop 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 shutdown_connections 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcd8100170 0x7fbcd8074af0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fbcc406c680 0x7fbcc406eb30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 --2- 172.21.15.153:0/165902536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd8101370 0x7fbcd8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 >> 172.21.15.153:0/165902536 conn(0x7fbcd80fb6e0 msgr2=0x7fbcd81045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 shutdown_connections 2024-05-06T14:30:27.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:27.576+0000 7fbcddf04700 1 -- 172.21.15.153:0/165902536 wait complete. 2024-05-06T14:30:28.136 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:28 smithi153 ceph-mon[27177]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-05-06T14:30:28.136 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:28.136 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:28 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:30:28.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T14:30:28.353 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/mon.smithi153/config 2024-05-06T14:30:28.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:28 smithi190 ceph-mon[33671]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-05-06T14:30:28.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:28.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:28 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:30:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:29 smithi190 ceph-mon[33671]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:30:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:29 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:30:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:29 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:30:29.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:29 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:29.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:29 smithi153 ceph-mon[27177]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:30:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:29 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:30:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:29 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:30:29.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:29 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:30.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:30 smithi153 ceph-mon[27177]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:30:30.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:30 smithi153 ceph-mon[27177]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-06T14:30:30.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:30 smithi190 ceph-mon[33671]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:30:30.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:30 smithi190 ceph-mon[33671]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 -- 172.21.15.153:0/2891437317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 msgr2=0x7f38e81017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 --2- 172.21.15.153:0/2891437317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e81017a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009910 tx=0x7f38d8009c20 comp rx=0 tx=0).stop 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 -- 172.21.15.153:0/2891437317 shutdown_connections 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 --2- 172.21.15.153:0/2891437317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e81017a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 --2- 172.21.15.153:0/2891437317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38e8100150 0x7f38e8100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 -- 172.21.15.153:0/2891437317 >> 172.21.15.153:0/2891437317 conn(0x7f38e80fb6e0 msgr2=0x7f38e80fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.248+0000 7f38ecb88700 1 -- 172.21.15.153:0/2891437317 shutdown_connections 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 -- 172.21.15.153:0/2891437317 wait complete. 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 Processor -- start 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 -- start start 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38e8100150 0x7f38e8074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e8073680 con 0x7f38e8101350 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.249+0000 7f38ecb88700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80737c0 con 0x7f38e8100150 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40754/0 (socket says 172.21.15.153:40754) 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 -- 172.21.15.153:0/492396892 learned_addr learned my addr 172.21.15.153:0/492396892 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e659c700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38e8100150 0x7f38e8074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 -- 172.21.15.153:0/492396892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38e8100150 msgr2=0x7f38e8074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38e8100150 0x7f38e8074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 -- 172.21.15.153:0/492396892 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d80095f0 con 0x7f38e8101350 2024-05-06T14:30:31.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38e5d9b700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e8073140 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f38d80092e0 tx=0x7f38d800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:31.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38d801d070 con 0x7f38e8101350 2024-05-06T14:30:31.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f38d800f460 con 0x7f38e8101350 2024-05-06T14:30:31.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e8073a40 con 0x7f38e8101350 2024-05-06T14:30:31.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38d8021620 con 0x7f38e8101350 2024-05-06T14:30:31.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.250+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e8073f30 con 0x7f38e8101350 2024-05-06T14:30:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.251+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e8108ca0 con 0x7f38e8101350 2024-05-06T14:30:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.252+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f38d800fa70 con 0x7f38e8101350 2024-05-06T14:30:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.252+0000 7f38d77fe700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f38d006c680 0x7f38d006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.252+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f38d808cb80 con 0x7f38e8101350 2024-05-06T14:30:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.252+0000 7f38e659c700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f38d006c680 0x7f38d006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.253+0000 7f38e659c700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f38d006c680 0x7f38d006eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38e81011b0 tx=0x7f38dc00b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:31.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.257+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f38d805ac90 con 0x7f38e8101350 2024-05-06T14:30:31.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.466+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f38e8105ca0 con 0x7f38d006c680 2024-05-06T14:30:31.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.467+0000 7f38d77fe700 1 -- 172.21.15.153:0/492396892 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f38e8105ca0 con 0x7f38d006c680 2024-05-06T14:30:31.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f38d006c680 msgr2=0x7f38d006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f38d006c680 0x7f38d006eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38e81011b0 tx=0x7f38dc00b410 comp rx=0 tx=0).stop 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 msgr2=0x7f38e8073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e8073140 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f38d80092e0 tx=0x7f38d800bd70 comp rx=0 tx=0).stop 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 shutdown_connections 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f38d006c680 0x7f38d006eb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38e8100150 0x7f38e8074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 --2- 172.21.15.153:0/492396892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e8101350 0x7f38e8073140 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.469+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 >> 172.21.15.153:0/492396892 conn(0x7f38e80fb6e0 msgr2=0x7f38e8104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.470+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 shutdown_connections 2024-05-06T14:30:31.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.470+0000 7f38ecb88700 1 -- 172.21.15.153:0/492396892 wait complete. 2024-05-06T14:30:31.467 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-05-06T14:30:31.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 -- 172.21.15.153:0/1621946246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4100190 msgr2=0x7f11d41005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 --2- 172.21.15.153:0/1621946246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4100190 0x7f11d41005a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009910 tx=0x7f11c4009c20 comp rx=0 tx=0).stop 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 -- 172.21.15.153:0/1621946246 shutdown_connections 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 --2- 172.21.15.153:0/1621946246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11d4101390 0x7f11d41017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 --2- 172.21.15.153:0/1621946246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4100190 0x7f11d41005a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 -- 172.21.15.153:0/1621946246 >> 172.21.15.153:0/1621946246 conn(0x7f11d40fb720 msgr2=0x7f11d40fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 -- 172.21.15.153:0/1621946246 shutdown_connections 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 -- 172.21.15.153:0/1621946246 wait complete. 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 Processor -- start 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.575+0000 7f11da216700 1 -- start start 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11da216700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11d4100190 0x7f11d4195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11da216700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 0x7f11d4195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11da216700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41964f0 con 0x7f11d4101390 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11da216700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4196630 con 0x7f11d4100190 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 0x7f11d4195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 0x7f11d4195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40766/0 (socket says 172.21.15.153:40766) 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 -- 172.21.15.153:0/2605713950 learned_addr learned my addr 172.21.15.153:0/2605713950 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11d37fe700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11d4100190 0x7f11d4195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 -- 172.21.15.153:0/2605713950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11d4100190 msgr2=0x7f11d4195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11d4100190 0x7f11d4195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 -- 172.21.15.153:0/2605713950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40095f0 con 0x7f11d4101390 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11caffd700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 0x7f11d4195ed0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00d8d0 tx=0x7f11bc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11bc0098a0 con 0x7f11d4101390 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11bc010460 con 0x7f11d4101390 2024-05-06T14:30:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d419b0e0 con 0x7f11d4101390 2024-05-06T14:30:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11bc00f5f0 con 0x7f11d4101390 2024-05-06T14:30:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.576+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d419b630 con 0x7f11d4101390 2024-05-06T14:30:31.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.577+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4108ce0 con 0x7f11d4101390 2024-05-06T14:30:31.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.578+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f11bc009a00 con 0x7f11d4101390 2024-05-06T14:30:31.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.578+0000 7f11d17fa700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f11c006c680 0x7f11c006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.578+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f11bc08b7c0 con 0x7f11d4101390 2024-05-06T14:30:31.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.578+0000 7f11d37fe700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f11c006c680 0x7f11c006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.579+0000 7f11d37fe700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f11c006c680 0x7f11c006eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f11c4006010 tx=0x7f11c400b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:31.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.583+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f11bc0598d0 con 0x7f11d4101390 2024-05-06T14:30:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.787+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f11d4105ce0 con 0x7f11c006c680 2024-05-06T14:30:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.788+0000 7f11d17fa700 1 -- 172.21.15.153:0/2605713950 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f11d4105ce0 con 0x7f11c006c680 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f11c006c680 msgr2=0x7f11c006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f11c006c680 0x7f11c006eb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f11c4006010 tx=0x7f11c400b540 comp rx=0 tx=0).stop 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 msgr2=0x7f11d4195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 0x7f11d4195ed0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00d8d0 tx=0x7f11bc00dbe0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 shutdown_connections 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f11c006c680 0x7f11c006eb30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11d4100190 0x7f11d4195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 --2- 172.21.15.153:0/2605713950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d4101390 0x7f11d4195ed0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 >> 172.21.15.153:0/2605713950 conn(0x7f11d40fb720 msgr2=0x7f11d41045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 shutdown_connections 2024-05-06T14:30:31.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.790+0000 7f11da216700 1 -- 172.21.15.153:0/2605713950 wait complete. 2024-05-06T14:30:31.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 -- 172.21.15.153:0/1872903616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 msgr2=0x7faee81018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 --2- 172.21.15.153:0/1872903616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee81018b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faee4009910 tx=0x7faee4009c20 comp rx=0 tx=0).stop 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 -- 172.21.15.153:0/1872903616 shutdown_connections 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 --2- 172.21.15.153:0/1872903616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee81018b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 --2- 172.21.15.153:0/1872903616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee80fe480 0x7faee80fe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 -- 172.21.15.153:0/1872903616 >> 172.21.15.153:0/1872903616 conn(0x7faee80f9dd0 msgr2=0x7faee80fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 -- 172.21.15.153:0/1872903616 shutdown_connections 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.894+0000 7faef0309700 1 -- 172.21.15.153:0/1872903616 wait complete. 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faef0309700 1 Processor -- start 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faef0309700 1 -- start start 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faef0309700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee80fe480 0x7faee8195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faef0309700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee8195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faef0309700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81965a0 con 0x7faee8101460 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faef0309700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81966e0 con 0x7faee80fe480 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faeed8a4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee8195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faeed8a4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee8195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40792/0 (socket says 172.21.15.153:40792) 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faeee0a5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee80fe480 0x7faee8195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.895+0000 7faeed8a4700 1 -- 172.21.15.153:0/1972862166 learned_addr learned my addr 172.21.15.153:0/1972862166 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faeed8a4700 1 -- 172.21.15.153:0/1972862166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee80fe480 msgr2=0x7faee8195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faeed8a4700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee80fe480 0x7faee8195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faeed8a4700 1 -- 172.21.15.153:0/1972862166 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee40095f0 con 0x7faee8101460 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faeed8a4700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee8195f80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7faee4006010 tx=0x7faee400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee401d070 con 0x7faee8101460 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee819b130 con 0x7faee8101460 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faee4022470 con 0x7faee8101460 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee400f690 con 0x7faee8101460 2024-05-06T14:30:31.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.896+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee819b620 con 0x7faee8101460 2024-05-06T14:30:31.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.897+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee8108d00 con 0x7faee8101460 2024-05-06T14:30:31.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.897+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7faee40225e0 con 0x7faee8101460 2024-05-06T14:30:31.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.898+0000 7faedf7fe700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7faed406c680 0x7faed406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:31.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.898+0000 7faeee0a5700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7faed406c680 0x7faed406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:31.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.898+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7faee408ce60 con 0x7faee8101460 2024-05-06T14:30:31.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.898+0000 7faeee0a5700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7faed406c680 0x7faed406eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faee8068b30 tx=0x7faed800a380 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:31.903+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7faee405b020 con 0x7faee8101460 2024-05-06T14:30:32.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (unknown) 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: Upgrade: Need to upgrade myself (mgr.smithi153.dolfed) 2024-05-06T14:30:32.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:32 smithi153 ceph-mon[27177]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb on smithi190 2024-05-06T14:30:32.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.105+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faee8105db0 con 0x7faed406c680 2024-05-06T14:30:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.115+0000 7faedf7fe700 1 -- 172.21.15.153:0/1972862166 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7faee8105db0 con 0x7faed406c680 2024-05-06T14:30:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:30:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 51s ago 6m 25.2M - 0.25.0 c8568f914cd2 5869e5f8b16f 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi153 smithi153 running (7m) 51s ago 7m 7415k - 18.2.0 dc2bc1663786 9903f06d2ec9 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi190 smithi190 running (5m) 55s ago 5m 7360k - 18.2.0 dc2bc1663786 8fa6eb9bb609 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (6m) 51s ago 6m 6660k - 18.2.0 dc2bc1663786 d9ddce025318 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi190 smithi190 running (5m) 55s ago 5m 6669k - 18.2.0 dc2bc1663786 1b44eaee674c 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 51s ago 6m 86.2M - 9.4.7 954c08fa6188 d15dfe6401b3 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.ftbnnk smithi153 running (71s) 51s ago 70s 18.0M - 18.2.0 dc2bc1663786 ecb0b5505691 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.hiwroq smithi153 running (78s) 51s ago 77s 11.4M - 18.2.0 dc2bc1663786 0c0228faf2c1 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.psnzbz smithi190 running (75s) 55s ago 75s 15.8M - 18.2.0 dc2bc1663786 ad189e8605bd 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.quxdec smithi190 running (68s) 55s ago 68s 15.5M - 18.2.0 dc2bc1663786 869fb60edc82 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.dolfed smithi153 *:9283,8765,8443 running (8m) 51s ago 8m 487M - 18.2.0 dc2bc1663786 0562ae98d3da 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi190.qauohf smithi190 *:8443,9283,8765 running (5m) 55s ago 5m 437M - 18.2.0 dc2bc1663786 1cd612378142 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (8m) 51s ago 8m 45.0M 2048M 18.2.0 dc2bc1663786 44f2993c90f4 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi190 smithi190 running (5m) 55s ago 5m 35.1M 2048M 18.2.0 dc2bc1663786 88aeab4dfa5e 2024-05-06T14:30:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 51s ago 6m 14.5M - 1.5.0 0da6a335fe13 5c569e0ae0c9 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi190 smithi190 *:9100 running (5m) 55s ago 5m 17.3M - 1.5.0 0da6a335fe13 d1be4aed7b03 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (4m) 51s ago 4m 48.2M 3946M 18.2.0 dc2bc1663786 6c202dfae018 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 51s ago 4m 48.7M 3946M 18.2.0 dc2bc1663786 78ff3749996c 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (3m) 51s ago 3m 49.6M 3946M 18.2.0 dc2bc1663786 3fbd486b6ac6 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi190 running (3m) 55s ago 3m 48.5M 4970M 18.2.0 dc2bc1663786 79af272339eb 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi190 running (3m) 55s ago 3m 44.1M 4970M 18.2.0 dc2bc1663786 6db95b0197f1 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi190 running (2m) 55s ago 2m 43.5M 4970M 18.2.0 dc2bc1663786 ae3651b2db8d 2024-05-06T14:30:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (4m) 51s ago 6m 63.3M - 2.43.0 a07b618ecd1d c8f2e7c500cf 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7faed406c680 msgr2=0x7faed406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7faed406c680 0x7faed406eb30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faee8068b30 tx=0x7faed800a380 comp rx=0 tx=0).stop 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 msgr2=0x7faee8195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee8195f80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7faee4006010 tx=0x7faee400bd70 comp rx=0 tx=0).stop 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 shutdown_connections 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7faed406c680 0x7faed406eb30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee80fe480 0x7faee8195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 --2- 172.21.15.153:0/1972862166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faee8101460 0x7faee8195f80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 >> 172.21.15.153:0/1972862166 conn(0x7faee80f9dd0 msgr2=0x7faee8104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:32.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 shutdown_connections 2024-05-06T14:30:32.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.117+0000 7faef0309700 1 -- 172.21.15.153:0/1972862166 wait complete. 2024-05-06T14:30:32.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.219+0000 7fa177641700 1 -- 172.21.15.153:0/2043754516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 msgr2=0x7fa1700ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.219+0000 7fa177641700 1 --2- 172.21.15.153:0/2043754516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa1700ffe90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa160009910 tx=0x7fa160009c20 comp rx=0 tx=0).stop 2024-05-06T14:30:32.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.219+0000 7fa177641700 1 -- 172.21.15.153:0/2043754516 shutdown_connections 2024-05-06T14:30:32.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.219+0000 7fa177641700 1 --2- 172.21.15.153:0/2043754516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa1700ffe90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.219+0000 7fa177641700 1 --2- 172.21.15.153:0/2043754516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1700fe840 0x7fa1700fec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.219+0000 7fa177641700 1 -- 172.21.15.153:0/2043754516 >> 172.21.15.153:0/2043754516 conn(0x7fa1700f9dd0 msgr2=0x7fa1700fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:32.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 -- 172.21.15.153:0/2043754516 shutdown_connections 2024-05-06T14:30:32.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 -- 172.21.15.153:0/2043754516 wait complete. 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 Processor -- start 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 -- start start 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1700fe840 0x7fa170195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa170195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170196590 con 0x7fa1700ffa40 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa177641700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1701966d0 con 0x7fa1700fe840 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.220+0000 7fa174bdc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa170195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa174bdc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa170195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40798/0 (socket says 172.21.15.153:40798) 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa174bdc700 1 -- 172.21.15.153:0/4233247930 learned_addr learned my addr 172.21.15.153:0/4233247930 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:30:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa1753dd700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1700fe840 0x7fa170195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:32.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa174bdc700 1 -- 172.21.15.153:0/4233247930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1700fe840 msgr2=0x7fa170195a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa174bdc700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1700fe840 0x7fa170195a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa174bdc700 1 -- 172.21.15.153:0/4233247930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1600095f0 con 0x7fa1700ffa40 2024-05-06T14:30:32.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa174bdc700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa170195f70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa160006010 tx=0x7fa16000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:32.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa16001d070 con 0x7fa1700ffa40 2024-05-06T14:30:32.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa16000f460 con 0x7fa1700ffa40 2024-05-06T14:30:32.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa160017630 con 0x7fa1700ffa40 2024-05-06T14:30:32.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17019b120 con 0x7fa1700ffa40 2024-05-06T14:30:32.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.221+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17019b610 con 0x7fa1700ffa40 2024-05-06T14:30:32.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.222+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa170106b00 con 0x7fa1700ffa40 2024-05-06T14:30:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.223+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fa16000f5d0 con 0x7fa1700ffa40 2024-05-06T14:30:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.223+0000 7fa1667fc700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa15c06c680 0x7fa15c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.223+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa16008cdc0 con 0x7fa1700ffa40 2024-05-06T14:30:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.223+0000 7fa1753dd700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa15c06c680 0x7fa15c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.224+0000 7fa1753dd700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa15c06c680 0x7fa15c06eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa1700ff8a0 tx=0x7fa16c00b510 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.227+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa16005aed0 con 0x7fa1700ffa40 2024-05-06T14:30:32.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-05-06T14:30:32.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:32.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (unknown) 2024-05-06T14:30:32.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:30:32.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:30:32.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:30:32.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: Upgrade: Need to upgrade myself (mgr.smithi153.dolfed) 2024-05-06T14:30:32.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:32 smithi190 ceph-mon[33671]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb on smithi190 2024-05-06T14:30:32.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.498+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa17019c050 con 0x7fa1700ffa40 2024-05-06T14:30:32.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.498+0000 7fa1667fc700 1 -- 172.21.15.153:0/4233247930 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fa16005aa60 con 0x7fa1700ffa40 2024-05-06T14:30:32.485 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-05-06T14:30:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-05-06T14:30:32.487 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:30:32.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.500+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa15c06c680 msgr2=0x7fa15c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.500+0000 7fa177641700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa15c06c680 0x7fa15c06eb30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa1700ff8a0 tx=0x7fa16c00b510 comp rx=0 tx=0).stop 2024-05-06T14:30:32.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 msgr2=0x7fa170195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa170195f70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa160006010 tx=0x7fa16000bd70 comp rx=0 tx=0).stop 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 shutdown_connections 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fa15c06c680 0x7fa15c06eb30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1700fe840 0x7fa170195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 --2- 172.21.15.153:0/4233247930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1700ffa40 0x7fa170195f70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 >> 172.21.15.153:0/4233247930 conn(0x7fa1700f9dd0 msgr2=0x7fa170102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 shutdown_connections 2024-05-06T14:30:32.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.501+0000 7fa177641700 1 -- 172.21.15.153:0/4233247930 wait complete. 2024-05-06T14:30:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 -- 172.21.15.153:0/2447428053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564103540 msgr2=0x7fd564105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 --2- 172.21.15.153:0/2447428053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564103540 0x7fd564105920 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd550009910 tx=0x7fd550009c20 comp rx=0 tx=0).stop 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 -- 172.21.15.153:0/2447428053 shutdown_connections 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 --2- 172.21.15.153:0/2447428053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564103540 0x7fd564105920 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 --2- 172.21.15.153:0/2447428053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd564100c20 0x7fd564103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 -- 172.21.15.153:0/2447428053 >> 172.21.15.153:0/2447428053 conn(0x7fd5640fa620 msgr2=0x7fd5640fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 -- 172.21.15.153:0/2447428053 shutdown_connections 2024-05-06T14:30:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 -- 172.21.15.153:0/2447428053 wait complete. 2024-05-06T14:30:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 Processor -- start 2024-05-06T14:30:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.607+0000 7fd56acf0700 1 -- start start 2024-05-06T14:30:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd56acf0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 0x7fd564195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd56acf0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd564103540 0x7fd564195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd56acf0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5641964b0 con 0x7fd564100c20 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd56acf0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5641965f0 con 0x7fd564103540 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 0x7fd564195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 0x7fd564195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40822/0 (socket says 172.21.15.153:40822) 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 -- 172.21.15.153:0/530408292 learned_addr learned my addr 172.21.15.153:0/530408292 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd563fff700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd564103540 0x7fd564195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 -- 172.21.15.153:0/530408292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd564103540 msgr2=0x7fd564195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd564103540 0x7fd564195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 -- 172.21.15.153:0/530408292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500095f0 con 0x7fd564100c20 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd568a8c700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 0x7fd564195950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd55800e920 tx=0x7fd55800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd55800c9d0 con 0x7fd564100c20 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd558010430 con 0x7fd564100c20 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56419b0a0 con 0x7fd564100c20 2024-05-06T14:30:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.608+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd558018650 con 0x7fd564100c20 2024-05-06T14:30:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.609+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56419b5f0 con 0x7fd564100c20 2024-05-06T14:30:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.610+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56418fbb0 con 0x7fd564100c20 2024-05-06T14:30:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.610+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd5580105a0 con 0x7fd564100c20 2024-05-06T14:30:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.610+0000 7fd561ffb700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd55406c680 0x7fd55406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:30:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.611+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd558014070 con 0x7fd564100c20 2024-05-06T14:30:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.611+0000 7fd563fff700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd55406c680 0x7fd55406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:30:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.611+0000 7fd563fff700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd55406c680 0x7fd55406eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd550009fd0 tx=0x7fd550005940 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:30:32.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.615+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd558051720 con 0x7fd564100c20 2024-05-06T14:30:32.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.824+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd564061190 con 0x7fd55406c680 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.825+0000 7fd561ffb700 1 -- 172.21.15.153:0/530408292 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fd564061190 con 0x7fd55406c680 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb", 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb image on host smithi190", 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-05-06T14:30:32.812 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd55406c680 msgr2=0x7fd55406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd55406c680 0x7fd55406eb30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd550009fd0 tx=0x7fd550005940 comp rx=0 tx=0).stop 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 msgr2=0x7fd564195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 0x7fd564195950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd55800e920 tx=0x7fd55800ec30 comp rx=0 tx=0).stop 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 shutdown_connections 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564100c20 0x7fd564195950 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd55406c680 0x7fd55406eb30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 --2- 172.21.15.153:0/530408292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd564103540 0x7fd564195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:30:32.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 >> 172.21.15.153:0/530408292 conn(0x7fd5640fa620 msgr2=0x7fd5640fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:30:32.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 shutdown_connections 2024-05-06T14:30:32.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:30:32.827+0000 7fd56acf0700 1 -- 172.21.15.153:0/530408292 wait complete. 2024-05-06T14:30:33.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:33 smithi153 ceph-mon[27177]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:33.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:33 smithi153 ceph-mon[27177]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:33 smithi153 ceph-mon[27177]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:33.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:33 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/4233247930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:30:33.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:33 smithi190 ceph-mon[33671]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:33.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:33 smithi190 ceph-mon[33671]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:33.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:33 smithi190 ceph-mon[33671]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:33.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:33 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/4233247930' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:30:34.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:34 smithi190 ceph-mon[33671]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:34.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:34 smithi190 ceph-mon[33671]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-05-06T14:30:34.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:34 smithi153 ceph-mon[27177]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:30:34.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:34 smithi153 ceph-mon[27177]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-05-06T14:30:36.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:36 smithi153 ceph-mon[27177]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-05-06T14:30:36.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:36 smithi190 ceph-mon[33671]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-05-06T14:30:38.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:38 smithi190 ceph-mon[33671]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-05-06T14:30:38.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:38 smithi153 ceph-mon[27177]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-05-06T14:30:40.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:40 smithi153 ceph-mon[27177]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-05-06T14:30:40.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:40 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:40.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:40 smithi190 ceph-mon[33671]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-05-06T14:30:40.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:40 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:42.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:42 smithi190 ceph-mon[33671]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-06T14:30:42.443 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:42 smithi153 ceph-mon[27177]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-06T14:30:44.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:44 smithi190 ceph-mon[33671]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-06T14:30:44.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:44 smithi153 ceph-mon[27177]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-05-06T14:30:46.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:46 smithi190 ceph-mon[33671]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:46.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:46 smithi153 ceph-mon[27177]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:48.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:48 smithi190 ceph-mon[33671]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:48.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:48 smithi153 ceph-mon[27177]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:50.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:50 smithi190 ceph-mon[33671]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:50.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:50 smithi153 ceph-mon[27177]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:52.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:52 smithi190 ceph-mon[33671]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:52.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:52 smithi153 ceph-mon[27177]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:54.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:54 smithi190 ceph-mon[33671]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:54.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:54 smithi153 ceph-mon[27177]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:55.431 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:55 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:55.437 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:55 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:30:56.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:56 smithi190 ceph-mon[33671]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:56.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:56 smithi153 ceph-mon[27177]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:58.185 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:30:58 smithi190 ceph-mon[33671]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:30:58.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:30:58 smithi153 ceph-mon[27177]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:00.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:00 smithi190 ceph-mon[33671]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:00.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:00 smithi153 ceph-mon[27177]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:02.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:02 smithi190 ceph-mon[33671]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:02.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:02 smithi153 ceph-mon[27177]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.992+0000 7f6b0759e700 1 -- 172.21.15.153:0/3230781793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 msgr2=0x7f6b080ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.992+0000 7f6b0759e700 1 --2- 172.21.15.153:0/3230781793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b080ff2a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009910 tx=0x7f6af8009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.993+0000 7f6b0759e700 1 -- 172.21.15.153:0/3230781793 shutdown_connections 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.993+0000 7f6b0759e700 1 --2- 172.21.15.153:0/3230781793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 0x7f6b08100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.993+0000 7f6b0759e700 1 --2- 172.21.15.153:0/3230781793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b080ff2a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.993+0000 7f6b0759e700 1 -- 172.21.15.153:0/3230781793 >> 172.21.15.153:0/3230781793 conn(0x7f6b080fa440 msgr2=0x7f6b080fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.993+0000 7f6b0759e700 1 -- 172.21.15.153:0/3230781793 shutdown_connections 2024-05-06T14:31:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.993+0000 7f6b0759e700 1 -- 172.21.15.153:0/3230781793 wait complete. 2024-05-06T14:31:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0759e700 1 Processor -- start 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0759e700 1 -- start start 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0759e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b08068f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0759e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 0x7f6b0819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0759e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0819aad0 con 0x7f6b080fee90 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0759e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0819ac40 con 0x7f6b080ffff0 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b08068f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b05d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 0x7f6b0819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b08068f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42218/0 (socket says 172.21.15.153:42218) 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.994+0000 7f6b0659c700 1 -- 172.21.15.153:0/2177267005 learned_addr learned my addr 172.21.15.153:0/2177267005 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6b05d9b700 1 -- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 msgr2=0x7f6b08068f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6b05d9b700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b08068f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6b05d9b700 1 -- 172.21.15.153:0/2177267005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af80095f0 con 0x7f6b080ffff0 2024-05-06T14:31:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6b05d9b700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 0x7f6b0819a500 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00e970 tx=0x7f6afc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6afc00c9d0 con 0x7f6b080ffff0 2024-05-06T14:31:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6afc010430 con 0x7f6b080ffff0 2024-05-06T14:31:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6afc018630 con 0x7f6b080ffff0 2024-05-06T14:31:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0819af20 con 0x7f6b080ffff0 2024-05-06T14:31:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.995+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b081942d0 con 0x7f6b080ffff0 2024-05-06T14:31:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.997+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b08107960 con 0x7f6b080ffff0 2024-05-06T14:31:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.998+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f6afc00cb30 con 0x7f6b080ffff0 2024-05-06T14:31:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.998+0000 7f6af77fe700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6af006c680 0x7f6af006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.998+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6afc014070 con 0x7f6b080ffff0 2024-05-06T14:31:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.999+0000 7f6b0659c700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6af006c680 0x7f6af006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:02.999+0000 7f6b0659c700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6af006c680 0x7f6af006eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009fd0 tx=0x7f6af8005a90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:02.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.006+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6afc051720 con 0x7f6b080ffff0 2024-05-06T14:31:03.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.423+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b081049d0 con 0x7f6af006c680 2024-05-06T14:31:03.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.425+0000 7f6af77fe700 1 -- 172.21.15.153:0/2177267005 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f6b081049d0 con 0x7f6af006c680 2024-05-06T14:31:03.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6af006c680 msgr2=0x7f6af006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:03.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6af006c680 0x7f6af006eb30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009fd0 tx=0x7f6af8005a90 comp rx=0 tx=0).stop 2024-05-06T14:31:03.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 msgr2=0x7f6b0819a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:03.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 0x7f6b0819a500 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00e970 tx=0x7f6afc00ec80 comp rx=0 tx=0).stop 2024-05-06T14:31:03.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 shutdown_connections 2024-05-06T14:31:03.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b080fee90 0x7f6b08068f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f6af006c680 0x7f6af006eb30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 --2- 172.21.15.153:0/2177267005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b080ffff0 0x7f6b0819a500 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.428+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 >> 172.21.15.153:0/2177267005 conn(0x7f6b080fa440 msgr2=0x7f6b081032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:03.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.429+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 shutdown_connections 2024-05-06T14:31:03.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.429+0000 7f6b0759e700 1 -- 172.21.15.153:0/2177267005 wait complete. 2024-05-06T14:31:03.431 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-05-06T14:31:03.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.568+0000 7f2323d32700 1 -- 172.21.15.153:0/182600142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 msgr2=0x7f231c0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:03.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.568+0000 7f2323d32700 1 --2- 172.21.15.153:0/182600142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c0ff980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2318009910 tx=0x7f2318009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:03.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.568+0000 7f2323d32700 1 -- 172.21.15.153:0/182600142 shutdown_connections 2024-05-06T14:31:03.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.568+0000 7f2323d32700 1 --2- 172.21.15.153:0/182600142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f231c100290 0x7f231c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.568+0000 7f2323d32700 1 --2- 172.21.15.153:0/182600142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c0ff980 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.568+0000 7f2323d32700 1 -- 172.21.15.153:0/182600142 >> 172.21.15.153:0/182600142 conn(0x7f231c0f9d80 msgr2=0x7f231c0fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:03.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.569+0000 7f2323d32700 1 -- 172.21.15.153:0/182600142 shutdown_connections 2024-05-06T14:31:03.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.569+0000 7f2323d32700 1 -- 172.21.15.153:0/182600142 wait complete. 2024-05-06T14:31:03.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2323d32700 1 Processor -- start 2024-05-06T14:31:03.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2323d32700 1 -- start start 2024-05-06T14:31:03.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2323d32700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2323d32700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f231c100290 0x7f231c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2323d32700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c1964f0 con 0x7f231c0ff570 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2323d32700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c196630 con 0x7f231c100290 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2321ace700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2321ace700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42246/0 (socket says 172.21.15.153:42246) 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.570+0000 7f2321ace700 1 -- 172.21.15.153:0/3473679732 learned_addr learned my addr 172.21.15.153:0/3473679732 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:03.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2321ace700 1 -- 172.21.15.153:0/3473679732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f231c100290 msgr2=0x7f231c195ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-06T14:31:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2321ace700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f231c100290 0x7f231c195ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2321ace700 1 -- 172.21.15.153:0/3473679732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180095f0 con 0x7f231c0ff570 2024-05-06T14:31:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2321ace700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c195990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f231800f690 tx=0x7f231800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231801d070 con 0x7f231c0ff570 2024-05-06T14:31:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c19b080 con 0x7f231c0ff570 2024-05-06T14:31:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2318005dc0 con 0x7f231c0ff570 2024-05-06T14:31:03.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.571+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c1057b0 con 0x7f231c0ff570 2024-05-06T14:31:03.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.572+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2318017630 con 0x7f231c0ff570 2024-05-06T14:31:03.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.574+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f231800fbd0 con 0x7f231c0ff570 2024-05-06T14:31:03.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.574+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2300005320 con 0x7f231c0ff570 2024-05-06T14:31:03.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.575+0000 7f2312ffd700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f230806c680 0x7f230806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.575+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f231808cdb0 con 0x7f231c0ff570 2024-05-06T14:31:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.575+0000 7f23212cd700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f230806c680 0x7f230806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.576+0000 7f23212cd700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f230806c680 0x7f230806eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f231c0ff2a0 tx=0x7f230c009260 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:03.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:03.584+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f231805aea0 con 0x7f231c0ff570 2024-05-06T14:31:03.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.004+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2300000bf0 con 0x7f230806c680 2024-05-06T14:31:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.005+0000 7f2312ffd700 1 -- 172.21.15.153:0/3473679732 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f2300000bf0 con 0x7f230806c680 2024-05-06T14:31:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.008+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f230806c680 msgr2=0x7f230806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.008+0000 7f2323d32700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f230806c680 0x7f230806eb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f231c0ff2a0 tx=0x7f230c009260 comp rx=0 tx=0).stop 2024-05-06T14:31:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.008+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 msgr2=0x7f231c195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.008+0000 7f2323d32700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c195990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f231800f690 tx=0x7f231800f770 comp rx=0 tx=0).stop 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 shutdown_connections 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c0ff570 0x7f231c195990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f230806c680 0x7f230806eb30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 --2- 172.21.15.153:0/3473679732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f231c100290 0x7f231c195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 >> 172.21.15.153:0/3473679732 conn(0x7f231c0f9d80 msgr2=0x7f231c1034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 shutdown_connections 2024-05-06T14:31:03.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.009+0000 7f2323d32700 1 -- 172.21.15.153:0/3473679732 wait complete. 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 -- 172.21.15.153:0/1778593478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c100250 msgr2=0x7f446c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 --2- 172.21.15.153:0/1778593478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c100250 0x7f446c1006a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4460009910 tx=0x7f4460009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 -- 172.21.15.153:0/1778593478 shutdown_connections 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 --2- 172.21.15.153:0/1778593478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c100250 0x7f446c1006a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 --2- 172.21.15.153:0/1778593478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c0ff050 0x7f446c0ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 -- 172.21.15.153:0/1778593478 >> 172.21.15.153:0/1778593478 conn(0x7f446c0fa5e0 msgr2=0x7f446c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 -- 172.21.15.153:0/1778593478 shutdown_connections 2024-05-06T14:31:04.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.174+0000 7f4470a8c700 1 -- 172.21.15.153:0/1778593478 wait complete. 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.175+0000 7f4470a8c700 1 Processor -- start 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.175+0000 7f4470a8c700 1 -- start start 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.175+0000 7f4470a8c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c0ff050 0x7f446c102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.175+0000 7f4470a8c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 0x7f446c101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.175+0000 7f4470a8c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c1995e0 con 0x7f446c0ff050 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.175+0000 7f4470a8c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c199720 con 0x7f446c100250 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 0x7f446c101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 0x7f446c101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:40656/0 (socket says 172.21.15.153:40656) 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 -- 172.21.15.153:0/3669915317 learned_addr learned my addr 172.21.15.153:0/3669915317 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 -- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c0ff050 msgr2=0x7f446c102fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c0ff050 0x7f446c102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 -- 172.21.15.153:0/3669915317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44600095f0 con 0x7f446c100250 2024-05-06T14:31:04.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4469d9b700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 0x7f446c101630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4460000c00 tx=0x7f446000f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446001d070 con 0x7f446c100250 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f44600052a0 con 0x7f446c100250 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44600226a0 con 0x7f446c100250 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.176+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c101b70 con 0x7f446c100250 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.177+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c101f70 con 0x7f446c100250 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.178+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c1021f0 con 0x7f446c100250 2024-05-06T14:31:04.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.178+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f4460003680 con 0x7f446c100250 2024-05-06T14:31:04.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.179+0000 7f445b7fe700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f445406c680 0x7f445406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:04.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.179+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f446008cf70 con 0x7f446c100250 2024-05-06T14:31:04.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.179+0000 7f446a59c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f445406c680 0x7f445406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:04.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.180+0000 7f446a59c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f445406c680 0x7f445406eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f446c1000b0 tx=0x7f445c0096c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:04.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.187+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f44600bf720 con 0x7f446c100250 2024-05-06T14:31:04.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:04 smithi153 ceph-mon[27177]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:04.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:04 smithi153 ceph-mon[27177]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:04.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:04 smithi190 ceph-mon[33671]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:04.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:04 smithi190 ceph-mon[33671]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:04.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.586+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f446c068ae0 con 0x7f445406c680 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.605+0000 7f445b7fe700 1 -- 172.21.15.153:0/3669915317 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f446c068ae0 con 0x7f445406c680 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 83s ago 7m 25.2M - 0.25.0 c8568f914cd2 5869e5f8b16f 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi153 smithi153 running (7m) 83s ago 7m 7415k - 18.2.0 dc2bc1663786 9903f06d2ec9 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi190 smithi190 running (6m) 87s ago 6m 7360k - 18.2.0 dc2bc1663786 8fa6eb9bb609 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (7m) 83s ago 7m 6660k - 18.2.0 dc2bc1663786 d9ddce025318 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi190 smithi190 running (6m) 87s ago 6m 6669k - 18.2.0 dc2bc1663786 1b44eaee674c 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 83s ago 6m 86.2M - 9.4.7 954c08fa6188 d15dfe6401b3 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.ftbnnk smithi153 running (103s) 83s ago 103s 18.0M - 18.2.0 dc2bc1663786 ecb0b5505691 2024-05-06T14:31:04.592 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.hiwroq smithi153 running (110s) 83s ago 110s 11.4M - 18.2.0 dc2bc1663786 0c0228faf2c1 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.psnzbz smithi190 running (107s) 87s ago 107s 15.8M - 18.2.0 dc2bc1663786 ad189e8605bd 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.quxdec smithi190 running (101s) 87s ago 100s 15.5M - 18.2.0 dc2bc1663786 869fb60edc82 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.dolfed smithi153 *:9283,8765,8443 running (8m) 83s ago 8m 487M - 18.2.0 dc2bc1663786 0562ae98d3da 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi190.qauohf smithi190 *:8443,9283,8765 running (6m) 87s ago 6m 437M - 18.2.0 dc2bc1663786 1cd612378142 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (8m) 83s ago 8m 45.0M 2048M 18.2.0 dc2bc1663786 44f2993c90f4 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi190 smithi190 running (6m) 87s ago 6m 35.1M 2048M 18.2.0 dc2bc1663786 88aeab4dfa5e 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 83s ago 7m 14.5M - 1.5.0 0da6a335fe13 5c569e0ae0c9 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi190 smithi190 *:9100 running (6m) 87s ago 6m 17.3M - 1.5.0 0da6a335fe13 d1be4aed7b03 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 83s ago 5m 48.2M 3946M 18.2.0 dc2bc1663786 6c202dfae018 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 83s ago 4m 48.7M 3946M 18.2.0 dc2bc1663786 78ff3749996c 2024-05-06T14:31:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 83s ago 4m 49.6M 3946M 18.2.0 dc2bc1663786 3fbd486b6ac6 2024-05-06T14:31:04.594 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi190 running (4m) 87s ago 4m 48.5M 4970M 18.2.0 dc2bc1663786 79af272339eb 2024-05-06T14:31:04.594 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi190 running (3m) 87s ago 3m 44.1M 4970M 18.2.0 dc2bc1663786 6db95b0197f1 2024-05-06T14:31:04.594 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi190 running (3m) 87s ago 3m 43.5M 4970M 18.2.0 dc2bc1663786 ae3651b2db8d 2024-05-06T14:31:04.594 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (5m) 83s ago 6m 63.3M - 2.43.0 a07b618ecd1d c8f2e7c500cf 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f445406c680 msgr2=0x7f445406eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f445406c680 0x7f445406eb30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f446c1000b0 tx=0x7f445c0096c0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 msgr2=0x7f446c101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 0x7f446c101630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4460000c00 tx=0x7f446000f740 comp rx=0 tx=0).stop 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 shutdown_connections 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f446c0ff050 0x7f446c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f445406c680 0x7f445406eb30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 --2- 172.21.15.153:0/3669915317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f446c100250 0x7f446c101630 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 >> 172.21.15.153:0/3669915317 conn(0x7f446c0fa5e0 msgr2=0x7f446c0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:04.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 shutdown_connections 2024-05-06T14:31:04.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.607+0000 7f4470a8c700 1 -- 172.21.15.153:0/3669915317 wait complete. 2024-05-06T14:31:04.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 -- 172.21.15.153:0/3276335792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748100c10 msgr2=0x7f4748102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 --2- 172.21.15.153:0/3276335792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748100c10 0x7f4748102ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f473c009910 tx=0x7f473c009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 -- 172.21.15.153:0/3276335792 shutdown_connections 2024-05-06T14:31:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 --2- 172.21.15.153:0/3276335792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4748103530 0x7f4748105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 --2- 172.21.15.153:0/3276335792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748100c10 0x7f4748102ff0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 -- 172.21.15.153:0/3276335792 >> 172.21.15.153:0/3276335792 conn(0x7f47480fa630 msgr2=0x7f47480fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 -- 172.21.15.153:0/3276335792 shutdown_connections 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.712+0000 7f474ead3700 1 -- 172.21.15.153:0/3276335792 wait complete. 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474ead3700 1 Processor -- start 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474ead3700 1 -- start start 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474ead3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4748100c10 0x7f47481959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474ead3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 0x7f4748195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474ead3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748196510 con 0x7f4748103530 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474ead3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748196650 con 0x7f4748100c10 2024-05-06T14:31:04.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f4747fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 0x7f4748195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f4747fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 0x7f4748195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42274/0 (socket says 172.21.15.153:42274) 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f4747fff700 1 -- 172.21.15.153:0/3625890402 learned_addr learned my addr 172.21.15.153:0/3625890402 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.713+0000 7f474c86f700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4748100c10 0x7f47481959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4747fff700 1 -- 172.21.15.153:0/3625890402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4748100c10 msgr2=0x7f47481959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4747fff700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4748100c10 0x7f47481959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4747fff700 1 -- 172.21.15.153:0/3625890402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0095f0 con 0x7f4748103530 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4747fff700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 0x7f4748195ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f473400d8d0 tx=0x7f473400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47340098a0 con 0x7f4748103530 2024-05-06T14:31:04.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4734010460 con 0x7f4748103530 2024-05-06T14:31:04.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f473400f5f0 con 0x7f4748103530 2024-05-06T14:31:04.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474819b0b0 con 0x7f4748103530 2024-05-06T14:31:04.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.714+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474819b600 con 0x7f4748103530 2024-05-06T14:31:04.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.716+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474818fba0 con 0x7f4748103530 2024-05-06T14:31:04.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.716+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7f4734009a00 con 0x7f4748103530 2024-05-06T14:31:04.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.717+0000 7f4745ffb700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f473806c680 0x7f473806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:04.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.717+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f473408b710 con 0x7f4748103530 2024-05-06T14:31:04.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.717+0000 7f474c86f700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f473806c680 0x7f473806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:04.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.718+0000 7f474c86f700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f473806c680 0x7f473806eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f473c006010 tx=0x7f473c00b540 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:04.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:04.722+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f47340598a0 con 0x7f4748103530 2024-05-06T14:31:04.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.002+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f474802cc30 con 0x7f4748103530 2024-05-06T14:31:04.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.003+0000 7f4745ffb700 1 -- 172.21.15.153:0/3625890402 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f4734059430 con 0x7f4748103530 2024-05-06T14:31:04.989 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:31:04.989 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-05-06T14:31:04.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-05-06T14:31:04.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:04.991 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-05-06T14:31:04.991 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 14 2024-05-06T14:31:04.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-05-06T14:31:04.991 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:31:04.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f473806c680 msgr2=0x7f473806eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f473806c680 0x7f473806eb30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f473c006010 tx=0x7f473c00b540 comp rx=0 tx=0).stop 2024-05-06T14:31:04.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 msgr2=0x7f4748195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 0x7f4748195ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f473400d8d0 tx=0x7f473400dbe0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 shutdown_connections 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7f473806c680 0x7f473806eb30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4748100c10 0x7f47481959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 --2- 172.21.15.153:0/3625890402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4748103530 0x7f4748195ef0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.006+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 >> 172.21.15.153:0/3625890402 conn(0x7f47480fa630 msgr2=0x7f47480fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.007+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 shutdown_connections 2024-05-06T14:31:04.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.007+0000 7f474ead3700 1 -- 172.21.15.153:0/3625890402 wait complete. 2024-05-06T14:31:05.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/2968380019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 msgr2=0x7fd9e40ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:05.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/2968380019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e40ff460 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009910 tx=0x7fd9d4009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:05.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/2968380019 shutdown_connections 2024-05-06T14:31:05.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/2968380019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e41006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/2968380019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e40ff460 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/2968380019 >> 172.21.15.153:0/2968380019 conn(0x7fd9e40fa5e0 msgr2=0x7fd9e40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/2968380019 shutdown_connections 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.119+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/2968380019 wait complete. 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.120+0000 7fd9e8fdf700 1 Processor -- start 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.120+0000 7fd9e8fdf700 1 -- start start 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e8fdf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e4195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e8fdf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e4195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e8fdf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e41964e0 con 0x7fd9e40ff050 2024-05-06T14:31:05.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e8fdf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4196620 con 0x7fd9e4100250 2024-05-06T14:31:05.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e4195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:05.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e1d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e4195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e4195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42302/0 (socket says 172.21.15.153:42302) 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e1d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e4195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:40688/0 (socket says 172.21.15.153:40688) 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e259c700 1 -- 172.21.15.153:0/831028692 learned_addr learned my addr 172.21.15.153:0/831028692 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e1d9b700 1 -- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 msgr2=0x7fd9e4195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e1d9b700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e4195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e1d9b700 1 -- 172.21.15.153:0/831028692 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40095f0 con 0x7fd9e4100250 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e259c700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e4195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.121+0000 7fd9e1d9b700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e4195ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800e970 tx=0x7fd9d800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.122+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9d800c9d0 con 0x7fd9e4100250 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.122+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e41014d0 con 0x7fd9e4100250 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.122+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e4101a20 con 0x7fd9e4100250 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.122+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd9d8010430 con 0x7fd9e4100250 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.122+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9d8018650 con 0x7fd9e4100250 2024-05-06T14:31:05.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.123+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c4005320 con 0x7fd9e4100250 2024-05-06T14:31:05.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.124+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 19) v1 ==== 89916+0+0 (secure 0 0 0) 0x7fd9d800cb30 con 0x7fd9e4100250 2024-05-06T14:31:05.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.124+0000 7fd9d37fe700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd9cc06c6d0 0x7fd9cc06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:05.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.124+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd9d8014070 con 0x7fd9e4100250 2024-05-06T14:31:05.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.124+0000 7fd9e259c700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd9cc06c6d0 0x7fd9cc06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:05.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.125+0000 7fd9e259c700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd9cc06c6d0 0x7fd9cc06eb80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400b6a0 tx=0x7fd9d400b5a0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:05.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.132+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd9d8051720 con 0x7fd9e4100250 2024-05-06T14:31:05.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:05 smithi153 ceph-mon[27177]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:05.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:05 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3625890402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:31:05.312 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:05 smithi190 ceph-mon[33671]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:05.312 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:05 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3625890402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:31:05.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.359+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 --> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9c4000bf0 con 0x7fd9cc06c6d0 2024-05-06T14:31:05.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.361+0000 7fd9d37fe700 1 -- 172.21.15.153:0/831028692 <== mgr.14217 v2:172.21.15.153:6800/1473246849 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fd9c4000bf0 con 0x7fd9cc06c6d0 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb", 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb image on host smithi190", 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-05-06T14:31:05.349 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:31:05.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd9cc06c6d0 msgr2=0x7fd9cc06eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:05.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd9cc06c6d0 0x7fd9cc06eb80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400b6a0 tx=0x7fd9d400b5a0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 msgr2=0x7fd9e4195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e4195ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800e970 tx=0x7fd9d800ec80 comp rx=0 tx=0).stop 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 shutdown_connections 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9e40ff050 0x7fd9e4195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.153:6800/1473246849,v1:172.21.15.153:6801/1473246849] conn(0x7fd9cc06c6d0 0x7fd9cc06eb80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 --2- 172.21.15.153:0/831028692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9e4100250 0x7fd9e4195ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.364+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 >> 172.21.15.153:0/831028692 conn(0x7fd9e40fa5e0 msgr2=0x7fd9e40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.365+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 shutdown_connections 2024-05-06T14:31:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:05.365+0000 7fd9e8fdf700 1 -- 172.21.15.153:0/831028692 wait complete. 2024-05-06T14:31:06.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:06 smithi190 ceph-mon[33671]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:06.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:06 smithi190 ceph-mon[33671]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:06.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:06 smithi190 ceph-mon[33671]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:06.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:06 smithi153 ceph-mon[27177]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:06.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:06 smithi153 ceph-mon[27177]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:06.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:06 smithi153 ceph-mon[27177]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:07.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:07 smithi190 ceph-mon[33671]: Upgrade: Updating mgr.smithi190.qauohf 2024-05-06T14:31:07.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:07.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.qauohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:31:07.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:31:07.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:07 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:31:07.436 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:07 smithi190 ceph-mon[33671]: Deploying daemon mgr.smithi190.qauohf on smithi190 2024-05-06T14:31:07.757 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:07 smithi153 ceph-mon[27177]: Upgrade: Updating mgr.smithi190.qauohf 2024-05-06T14:31:07.758 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:07.758 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi190.qauohf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:31:07.758 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:31:07.758 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:07 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:31:07.758 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:07 smithi153 ceph-mon[27177]: Deploying daemon mgr.smithi190.qauohf on smithi190 2024-05-06T14:31:08.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:08 smithi190 ceph-mon[33671]: pgmap v236: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:08.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:08 smithi153 ceph-mon[27177]: pgmap v236: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:10.636 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:10 smithi190 ceph-mon[33671]: pgmap v237: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:10.636 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:10 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:10.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:10 smithi153 ceph-mon[27177]: pgmap v237: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:10.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:10 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:11.865 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:11.865 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:11.865 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:11 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:11.865 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:11 smithi190 ceph-mon[33671]: pgmap v238: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:12.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:12.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:12.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:11 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:12.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:11 smithi153 ceph-mon[27177]: pgmap v238: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:14.044 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:14 smithi190 ceph-mon[33671]: pgmap v239: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:14.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:14 smithi153 ceph-mon[27177]: pgmap v239: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:15.164 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:15.165 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:15.165 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:15 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:15.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:15.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:15.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:15 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:16.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:16 smithi190 ceph-mon[33671]: pgmap v240: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:16.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:16 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:16.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:16 smithi153 ceph-mon[27177]: pgmap v240: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:16.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:16 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:18.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:18 smithi190 ceph-mon[33671]: pgmap v241: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:18 smithi153 ceph-mon[27177]: pgmap v241: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr:Note: switching to '70ce9539b62a15b9de429c1f0e91a1e012b099aa'. 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr:state without impacting any branches by switching back to a branch. 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.155 INFO:tasks.workunit.client.0.smithi153.stderr: git switch -c 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr:Or undo this operation with: 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr: git switch - 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-05-06T14:31:19.156 INFO:tasks.workunit.client.0.smithi153.stderr:HEAD is now at 70ce9539b62 Merge pull request #56964 from adk3798/squid-dashboard-anon-grafana 2024-05-06T14:31:19.160 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-06T14:31:19.197 INFO:tasks.workunit.client.0.smithi153.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-06T14:31:19.199 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-06T14:31:19.201 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-06T14:31:19.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:19.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:19.534 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr fail", "who": "smithi153.dolfed"}]: dispatch 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: osdmap e48: 6 total, 6 up, 6 in 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "mgr fail", "who": "smithi153.dolfed"}]': finished 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: mgrmap e20: smithi190.qauohf(active, starting, since 0.00714283s) 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: Active manager daemon smithi190.qauohf restarted 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: Activating manager daemon smithi190.qauohf 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: osdmap e49: 6 total, 6 up, 6 in 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: mgrmap e21: smithi190.qauohf(active, starting, since 0.00551047s) 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.? 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/crt"}]: dispatch 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/crt"}]: dispatch 2024-05-06T14:31:19.535 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ftbnnk"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.hiwroq"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.psnzbz"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.quxdec"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mgr metadata", "who": "smithi190.qauohf", "id": "smithi190.qauohf"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T14:31:19.536 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T14:31:19.537 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T14:31:19.537 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T14:31:19.537 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/key"}]: dispatch 2024-05-06T14:31:19.537 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T14:31:19.537 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:19 smithi190 ceph-mon[33671]: Manager daemon smithi190.qauohf is now available 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "mgr fail", "who": "smithi153.dolfed"}]: dispatch 2024-05-06T14:31:19.694 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: osdmap e48: 6 total, 6 up, 6 in 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.14217 172.21.15.153:0/1316692429' entity='mgr.smithi153.dolfed' cmd='[{"prefix": "mgr fail", "who": "smithi153.dolfed"}]': finished 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: mgrmap e20: smithi190.qauohf(active, starting, since 0.00714283s) 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: Active manager daemon smithi190.qauohf restarted 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: Activating manager daemon smithi190.qauohf 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: osdmap e49: 6 total, 6 up, 6 in 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: mgrmap e21: smithi190.qauohf(active, starting, since 0.00551047s) 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/crt"}]: dispatch 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/crt"}]: dispatch 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mon metadata", "id": "smithi190"}]: dispatch 2024-05-06T14:31:19.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.ftbnnk"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.hiwroq"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.psnzbz"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi190.quxdec"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mgr metadata", "who": "smithi190.qauohf", "id": "smithi190.qauohf"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T14:31:19.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi190.qauohf/key"}]: dispatch 2024-05-06T14:31:19.697 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T14:31:19.697 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:19 smithi153 ceph-mon[27177]: Manager daemon smithi190.qauohf is now available 2024-05-06T14:31:19.833 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-06T14:31:19.885 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-06T14:31:19.943 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-06T14:31:19.945 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-06T14:31:19.945 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-06T14:31:19.994 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-06T14:31:19.997 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:31:19.997 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-06T14:31:20.053 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-05-06T14:31:20.055 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-05-06T14:31:20.055 DEBUG:teuthology.orchestra.run.smithi153:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=70ce9539b62a15b9de429c1f0e91a1e012b099aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-05-06T14:31:20.114 INFO:tasks.workunit.client.0.smithi153.stderr:+ mkdir -p fsstress 2024-05-06T14:31:20.116 INFO:tasks.workunit.client.0.smithi153.stderr:+ pushd fsstress 2024-05-06T14:31:20.116 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-05-06T14:31:20.116 INFO:tasks.workunit.client.0.smithi153.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-05-06T14:31:20.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:20 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:20.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:20 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.qauohf/mirror_snapshot_schedule"}]: dispatch 2024-05-06T14:31:20.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:20 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:20.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:20 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.qauohf/trash_purge_schedule"}]: dispatch 2024-05-06T14:31:20.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:20 smithi153 ceph-mon[27177]: mgrmap e22: smithi190.qauohf(active, since 1.00815s) 2024-05-06T14:31:20.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:20 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:20.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:20 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.qauohf/mirror_snapshot_schedule"}]: dispatch 2024-05-06T14:31:20.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:20 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:20.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:20 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi190.qauohf/trash_purge_schedule"}]: dispatch 2024-05-06T14:31:20.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:20 smithi190 ceph-mon[33671]: mgrmap e22: smithi190.qauohf(active, since 1.00815s) 2024-05-06T14:31:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:21 smithi153 ceph-mon[27177]: Deploying cephadm binary to smithi153 2024-05-06T14:31:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:21 smithi153 ceph-mon[27177]: Deploying cephadm binary to smithi190 2024-05-06T14:31:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:21 smithi153 ceph-mon[27177]: pgmap v3: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:21.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:21 smithi190 ceph-mon[33671]: Deploying cephadm binary to smithi153 2024-05-06T14:31:21.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:21 smithi190 ceph-mon[33671]: Deploying cephadm binary to smithi190 2024-05-06T14:31:21.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:21 smithi190 ceph-mon[33671]: pgmap v3: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:22.581 INFO:tasks.workunit.client.0.smithi153.stderr:+ tar xzf ltp-full.tgz 2024-05-06T14:31:22.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:22 smithi153 ceph-mon[27177]: [06/May/2024:14:31:20] ENGINE Bus STARTING 2024-05-06T14:31:22.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:22 smithi153 ceph-mon[27177]: [06/May/2024:14:31:20] ENGINE Serving on https://172.21.15.190:7150 2024-05-06T14:31:22.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:22 smithi153 ceph-mon[27177]: [06/May/2024:14:31:20] ENGINE Serving on http://172.21.15.190:8765 2024-05-06T14:31:22.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:22 smithi153 ceph-mon[27177]: [06/May/2024:14:31:20] ENGINE Bus STARTED 2024-05-06T14:31:22.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:22 smithi153 ceph-mon[27177]: pgmap v4: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:22.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:22 smithi153 ceph-mon[27177]: mgrmap e23: smithi190.qauohf(active, since 2s) 2024-05-06T14:31:22.836 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:22 smithi190 ceph-mon[33671]: [06/May/2024:14:31:20] ENGINE Bus STARTING 2024-05-06T14:31:22.836 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:22 smithi190 ceph-mon[33671]: [06/May/2024:14:31:20] ENGINE Serving on https://172.21.15.190:7150 2024-05-06T14:31:22.837 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:22 smithi190 ceph-mon[33671]: [06/May/2024:14:31:20] ENGINE Serving on http://172.21.15.190:8765 2024-05-06T14:31:22.837 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:22 smithi190 ceph-mon[33671]: [06/May/2024:14:31:20] ENGINE Bus STARTED 2024-05-06T14:31:22.837 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:22 smithi190 ceph-mon[33671]: pgmap v4: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:22.837 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:22 smithi190 ceph-mon[33671]: mgrmap e23: smithi190.qauohf(active, since 2s) 2024-05-06T14:31:24.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:24 smithi153 ceph-mon[27177]: pgmap v5: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:24.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:24 smithi153 ceph-mon[27177]: mgrmap e24: smithi190.qauohf(active, since 4s) 2024-05-06T14:31:24.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:24 smithi190 ceph-mon[33671]: pgmap v5: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:24.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:24 smithi190 ceph-mon[33671]: mgrmap e24: smithi190.qauohf(active, since 4s) 2024-05-06T14:31:25.336 INFO:tasks.workunit.client.1.smithi190.stderr:Note: switching to '70ce9539b62a15b9de429c1f0e91a1e012b099aa'. 2024-05-06T14:31:25.336 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.336 INFO:tasks.workunit.client.1.smithi190.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr:state without impacting any branches by switching back to a branch. 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr: git switch -c 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr:Or undo this operation with: 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.337 INFO:tasks.workunit.client.1.smithi190.stderr: git switch - 2024-05-06T14:31:25.338 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.338 INFO:tasks.workunit.client.1.smithi190.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-06T14:31:25.338 INFO:tasks.workunit.client.1.smithi190.stderr: 2024-05-06T14:31:25.338 INFO:tasks.workunit.client.1.smithi190.stderr:HEAD is now at 70ce9539b62 Merge pull request #56964 from adk3798/squid-dashboard-anon-grafana 2024-05-06T14:31:25.343 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-05-06T14:31:25.420 INFO:tasks.workunit.client.1.smithi190.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-06T14:31:25.424 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-05-06T14:31:25.424 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-06T14:31:26.644 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-06T14:31:26.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:26 smithi153 ceph-mon[27177]: pgmap v6: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:26.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:26 smithi153 ceph-mon[27177]: Standby manager daemon smithi153.dolfed started 2024-05-06T14:31:26.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:26 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.dolfed/crt"}]: dispatch 2024-05-06T14:31:26.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:26 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T14:31:26.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:26 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.dolfed/key"}]: dispatch 2024-05-06T14:31:26.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:26 smithi153 ceph-mon[27177]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T14:31:26.902 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-06T14:31:26.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:26 smithi190 ceph-mon[33671]: pgmap v6: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-06T14:31:26.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:26 smithi190 ceph-mon[33671]: Standby manager daemon smithi153.dolfed started 2024-05-06T14:31:26.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:26 smithi190 ceph-mon[33671]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.dolfed/crt"}]: dispatch 2024-05-06T14:31:26.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:26 smithi190 ceph-mon[33671]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T14:31:26.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:26 smithi190 ceph-mon[33671]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.dolfed/key"}]: dispatch 2024-05-06T14:31:26.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:26 smithi190 ceph-mon[33671]: from='mgr.? 172.21.15.153:0/1384776773' entity='mgr.smithi153.dolfed' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T14:31:27.058 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-05-06T14:31:27.060 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-05-06T14:31:27.060 INFO:tasks.workunit.client.1.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-06T14:31:27.146 INFO:tasks.workunit.client.1.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-05-06T14:31:27.149 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:31:27.150 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-05-06T14:31:27.207 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-05-06T14:31:27.209 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-05-06T14:31:27.209 DEBUG:teuthology.orchestra.run.smithi190:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=70ce9539b62a15b9de429c1f0e91a1e012b099aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-05-06T14:31:27.271 INFO:tasks.workunit.client.1.smithi190.stderr:+ mkdir -p fsstress 2024-05-06T14:31:27.275 INFO:tasks.workunit.client.1.smithi190.stderr:+ pushd fsstress 2024-05-06T14:31:27.275 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-05-06T14:31:27.276 INFO:tasks.workunit.client.1.smithi190.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-05-06T14:31:27.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:27 smithi153 ceph-mon[27177]: mgrmap e25: smithi190.qauohf(active, since 7s), standbys: smithi153.dolfed 2024-05-06T14:31:27.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:27 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mgr metadata", "who": "smithi153.dolfed", "id": "smithi153.dolfed"}]: dispatch 2024-05-06T14:31:27.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:27 smithi190 ceph-mon[33671]: mgrmap e25: smithi190.qauohf(active, since 7s), standbys: smithi153.dolfed 2024-05-06T14:31:27.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:27 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mgr metadata", "who": "smithi153.dolfed", "id": "smithi153.dolfed"}]: dispatch 2024-05-06T14:31:28.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:28 smithi190 ceph-mon[33671]: pgmap v7: 65 pgs: 65 active+clean; 11 MiB data, 178 MiB used, 536 GiB / 536 GiB avail; 29 KiB/s rd, 1.5 MiB/s wr, 60 op/s 2024-05-06T14:31:28.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:28 smithi153 ceph-mon[27177]: pgmap v7: 65 pgs: 65 active+clean; 11 MiB data, 178 MiB used, 536 GiB / 536 GiB avail; 29 KiB/s rd, 1.5 MiB/s wr, 60 op/s 2024-05-06T14:31:29.843 INFO:tasks.workunit.client.1.smithi190.stderr:+ tar xzf ltp-full.tgz 2024-05-06T14:31:30.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:30 smithi153 ceph-mon[27177]: pgmap v8: 65 pgs: 65 active+clean; 11 MiB data, 178 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-05-06T14:31:30.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:30 smithi190 ceph-mon[33671]: pgmap v8: 65 pgs: 65 active+clean; 11 MiB data, 178 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-05-06T14:31:32.608 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:32 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:32.608 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:32 smithi153 ceph-mon[27177]: pgmap v9: 65 pgs: 65 active+clean; 39 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 3.7 MiB/s wr, 289 op/s 2024-05-06T14:31:32.608 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:32 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:32.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:32 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:32.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:32 smithi190 ceph-mon[33671]: pgmap v9: 65 pgs: 65 active+clean; 39 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 3.7 MiB/s wr, 289 op/s 2024-05-06T14:31:32.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:32 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:33.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:33.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:33.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:33.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:33.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:33.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi153 to 1216M 2024-05-06T14:31:33.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:33.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:33 smithi190 ceph-mon[33671]: pgmap v10: 65 pgs: 65 active+clean; 39 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-05-06T14:31:34.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:34.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:34.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:34.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:34.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:34.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi153 to 1216M 2024-05-06T14:31:34.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:34.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:33 smithi153 ceph-mon[27177]: pgmap v10: 65 pgs: 65 active+clean; 39 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-05-06T14:31:35.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:35 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:35.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:35 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:35.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 -- 172.21.15.153:0/504097134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358101460 msgr2=0x7f23581018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:35.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 --2- 172.21.15.153:0/504097134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358101460 0x7f23581018b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2348009910 tx=0x7f2348009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:35.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 -- 172.21.15.153:0/504097134 shutdown_connections 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 --2- 172.21.15.153:0/504097134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358101460 0x7f23581018b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 --2- 172.21.15.153:0/504097134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23580fe480 0x7f23580fe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 -- 172.21.15.153:0/504097134 >> 172.21.15.153:0/504097134 conn(0x7f23580f9dd0 msgr2=0x7f23580fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 -- 172.21.15.153:0/504097134 shutdown_connections 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.497+0000 7f235fc53700 1 -- 172.21.15.153:0/504097134 wait complete. 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235fc53700 1 Processor -- start 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235fc53700 1 -- start start 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235fc53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 0x7f2358195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235fc53700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358101460 0x7f2358195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235fc53700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358196580 con 0x7f23580fe480 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235fc53700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23581966c0 con 0x7f2358101460 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d9ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 0x7f2358195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:35.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d9ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 0x7f2358195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53472/0 (socket says 172.21.15.153:53472) 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d9ef700 1 -- 172.21.15.153:0/2237687189 learned_addr learned my addr 172.21.15.153:0/2237687189 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d1ee700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358101460 0x7f2358195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d9ef700 1 -- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358101460 msgr2=0x7f2358195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d9ef700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358101460 0x7f2358195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.498+0000 7f235d9ef700 1 -- 172.21.15.153:0/2237687189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480095f0 con 0x7f23580fe480 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.499+0000 7f235d9ef700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 0x7f2358195a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f235400d900 tx=0x7f235400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.499+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f235400b920 con 0x7f23580fe480 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.499+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2354010460 con 0x7f23580fe480 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.499+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235819b170 con 0x7f23580fe480 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.499+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f235400f590 con 0x7f23580fe480 2024-05-06T14:31:35.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.499+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235819b6c0 con 0x7f23580fe480 2024-05-06T14:31:35.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.500+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2358108d00 con 0x7f23580fe480 2024-05-06T14:31:35.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.500+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f235400ba80 con 0x7f23580fe480 2024-05-06T14:31:35.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.501+0000 7f234effd700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f234406d3c0 0x7f234406f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:35.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.501+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f235408c8a0 con 0x7f23580fe480 2024-05-06T14:31:35.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.501+0000 7f235d1ee700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f234406d3c0 0x7f234406f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:35.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.502+0000 7f235d1ee700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f234406d3c0 0x7f234406f870 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2348005090 tx=0x7f234801a040 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:35.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.507+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f2354059b90 con 0x7f23580fe480 2024-05-06T14:31:35.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.792+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2358105db0 con 0x7f234406d3c0 2024-05-06T14:31:35.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.794+0000 7f234effd700 1 -- 172.21.15.153:0/2237687189 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f2358105db0 con 0x7f234406d3c0 2024-05-06T14:31:35.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f234406d3c0 msgr2=0x7f234406f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f234406d3c0 0x7f234406f870 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2348005090 tx=0x7f234801a040 comp rx=0 tx=0).stop 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 msgr2=0x7f2358195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 0x7f2358195a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f235400d900 tx=0x7f235400dc10 comp rx=0 tx=0).stop 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 shutdown_connections 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f234406d3c0 0x7f234406f870 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23580fe480 0x7f2358195a20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 --2- 172.21.15.153:0/2237687189 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2358101460 0x7f2358195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 >> 172.21.15.153:0/2237687189 conn(0x7f23580f9dd0 msgr2=0x7f2358104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 shutdown_connections 2024-05-06T14:31:35.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.797+0000 7f235fc53700 1 -- 172.21.15.153:0/2237687189 wait complete. 2024-05-06T14:31:35.796 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-05-06T14:31:35.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 -- 172.21.15.153:0/542225313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc0ff050 msgr2=0x7fc0dc0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:35.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/542225313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc0ff460 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009910 tx=0x7fc0cc009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 -- 172.21.15.153:0/542225313 shutdown_connections 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/542225313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc100250 0x7fc0dc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/542225313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc0ff460 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 -- 172.21.15.153:0/542225313 >> 172.21.15.153:0/542225313 conn(0x7fc0dc0fa5e0 msgr2=0x7fc0dc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 -- 172.21.15.153:0/542225313 shutdown_connections 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.915+0000 7fc0e13e7700 1 -- 172.21.15.153:0/542225313 wait complete. 2024-05-06T14:31:35.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.916+0000 7fc0e13e7700 1 Processor -- start 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.916+0000 7fc0e13e7700 1 -- start start 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.916+0000 7fc0e13e7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0e13e7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 0x7fc0dc194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0e13e7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc1953f0 con 0x7fc0dc100250 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0e13e7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc195530 con 0x7fc0dc0ff050 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0da7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 0x7fc0dc194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0da7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 0x7fc0dc194dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53494/0 (socket says 172.21.15.153:53494) 2024-05-06T14:31:35.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0da7fc700 1 -- 172.21.15.153:0/1188935274 learned_addr learned my addr 172.21.15.153:0/1188935274 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0daffd700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0daffd700 1 -- 172.21.15.153:0/1188935274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 msgr2=0x7fc0dc194dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0daffd700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 0x7fc0dc194dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0daffd700 1 -- 172.21.15.153:0/1188935274 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0095f0 con 0x7fc0dc0ff050 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0da7fc700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 0x7fc0dc194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.917+0000 7fc0daffd700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc194890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc000c00 tx=0x7fc0cc01a850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.918+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0cc01f070 con 0x7fc0dc0ff050 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.918+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc0683d0 con 0x7fc0dc0ff050 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.918+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0dc0688c0 con 0x7fc0dc0ff050 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.918+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc0cc005dc0 con 0x7fc0dc0ff050 2024-05-06T14:31:35.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.918+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0cc00f460 con 0x7fc0dc0ff050 2024-05-06T14:31:35.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.919+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c8005320 con 0x7fc0dc0ff050 2024-05-06T14:31:35.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.919+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7fc0cc023070 con 0x7fc0dc0ff050 2024-05-06T14:31:35.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.920+0000 7fc0c3fff700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fc0c406d3c0 0x7fc0c406f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:35.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.920+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fc0cc08f630 con 0x7fc0dc0ff050 2024-05-06T14:31:35.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.920+0000 7fc0da7fc700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fc0c406d3c0 0x7fc0c406f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:35.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.921+0000 7fc0da7fc700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fc0c406d3c0 0x7fc0c406f870 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0011550 tx=0x7fc0d0009260 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:35.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:35.925+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7fc0cc05c9d0 con 0x7fc0dc0ff050 2024-05-06T14:31:36.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.218+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0c8000bf0 con 0x7fc0c406d3c0 2024-05-06T14:31:36.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.220+0000 7fc0c3fff700 1 -- 172.21.15.153:0/1188935274 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fc0c8000bf0 con 0x7fc0c406d3c0 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fc0c406d3c0 msgr2=0x7fc0c406f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fc0c406d3c0 0x7fc0c406f870 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0011550 tx=0x7fc0d0009260 comp rx=0 tx=0).stop 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc0ff050 msgr2=0x7fc0dc194890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc194890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc000c00 tx=0x7fc0cc01a850 comp rx=0 tx=0).stop 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 shutdown_connections 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fc0c406d3c0 0x7fc0c406f870 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0dc0ff050 0x7fc0dc194890 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 --2- 172.21.15.153:0/1188935274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0dc100250 0x7fc0dc194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 >> 172.21.15.153:0/1188935274 conn(0x7fc0dc0fa5e0 msgr2=0x7fc0dc103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:36.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 shutdown_connections 2024-05-06T14:31:36.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.222+0000 7fc0e13e7700 1 -- 172.21.15.153:0/1188935274 wait complete. 2024-05-06T14:31:36.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 -- 172.21.15.153:0/410322086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8103530 msgr2=0x7f27c8105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 --2- 172.21.15.153:0/410322086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8103530 0x7f27c8105910 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f27b8009910 tx=0x7f27b8009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:36.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 -- 172.21.15.153:0/410322086 shutdown_connections 2024-05-06T14:31:36.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 --2- 172.21.15.153:0/410322086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8103530 0x7f27c8105910 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 --2- 172.21.15.153:0/410322086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c8100c10 0x7f27c8102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 -- 172.21.15.153:0/410322086 >> 172.21.15.153:0/410322086 conn(0x7f27c80fa630 msgr2=0x7f27c80fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:36.361 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:36 smithi153 ceph-mon[27177]: pgmap v11: 65 pgs: 65 active+clean; 39 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-05-06T14:31:36.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 -- 172.21.15.153:0/410322086 shutdown_connections 2024-05-06T14:31:36.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.373+0000 7f27d0011700 1 -- 172.21.15.153:0/410322086 wait complete. 2024-05-06T14:31:36.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27d0011700 1 Processor -- start 2024-05-06T14:31:36.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27d0011700 1 -- start start 2024-05-06T14:31:36.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27d0011700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 0x7f27c8100570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27d0011700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c8103530 0x7f27c80febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27d0011700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80ff100 con 0x7f27c8100c10 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27d0011700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c80ff240 con 0x7f27c8103530 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27cddad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 0x7f27c8100570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27cddad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 0x7f27c8100570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53506/0 (socket says 172.21.15.153:53506) 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.374+0000 7f27cddad700 1 -- 172.21.15.153:0/1889240469 learned_addr learned my addr 172.21.15.153:0/1889240469 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27cd5ac700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c8103530 0x7f27c80febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27cddad700 1 -- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c8103530 msgr2=0x7f27c80febc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27cddad700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c8103530 0x7f27c80febc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27cddad700 1 -- 172.21.15.153:0/1889240469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b80095f0 con 0x7f27c8100c10 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27cddad700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 0x7f27c8100570 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f27c400d8d0 tx=0x7f27c400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27c40098a0 con 0x7f27c8100c10 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c80ff520 con 0x7f27c8100c10 2024-05-06T14:31:36.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c819d9a0 con 0x7f27c8100c10 2024-05-06T14:31:36.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f27c4010460 con 0x7f27c8100c10 2024-05-06T14:31:36.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.375+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27c400f5f0 con 0x7f27c8100c10 2024-05-06T14:31:36.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.376+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ac005320 con 0x7f27c8100c10 2024-05-06T14:31:36.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.377+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f27c4009a00 con 0x7f27c8100c10 2024-05-06T14:31:36.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.378+0000 7f27beffd700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f27b406d3b0 0x7f27b406f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:36.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.378+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f27c408ca70 con 0x7f27c8100c10 2024-05-06T14:31:36.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.378+0000 7f27cd5ac700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f27b406d3b0 0x7f27b406f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:36.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.379+0000 7f27cd5ac700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f27b406d3b0 0x7f27b406f860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f27b8006010 tx=0x7f27b800b560 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:36.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.383+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f27c4059d60 con 0x7f27c8100c10 2024-05-06T14:31:36.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:36 smithi190 ceph-mon[33671]: pgmap v11: 65 pgs: 65 active+clean; 39 MiB data, 298 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-05-06T14:31:36.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.636+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f27ac000bf0 con 0x7f27b406d3b0 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.648+0000 7f27beffd700 1 -- 172.21.15.153:0/1889240469 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f27ac000bf0 con 0x7f27b406d3b0 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (6m) 5s ago 7m 24.9M - 0.25.0 c8568f914cd2 5869e5f8b16f 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi153 smithi153 running (8m) 5s ago 8m 7566k - 18.2.0 dc2bc1663786 9903f06d2ec9 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi190 smithi190 running (7m) 22s ago 7m 7512k - 18.2.0 dc2bc1663786 8fa6eb9bb609 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (8m) 5s ago 8m 6660k - 18.2.0 dc2bc1663786 d9ddce025318 2024-05-06T14:31:36.635 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi190 smithi190 running (6m) 22s ago 6m 6669k - 18.2.0 dc2bc1663786 1b44eaee674c 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (6m) 5s ago 7m 87.8M - 9.4.7 954c08fa6188 d15dfe6401b3 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.ftbnnk smithi153 running (2m) 5s ago 2m 81.3M - 18.2.0 dc2bc1663786 ecb0b5505691 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.hiwroq smithi153 running (2m) 5s ago 2m 12.8M - 18.2.0 dc2bc1663786 0c0228faf2c1 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.psnzbz smithi190 running (2m) 22s ago 2m 18.6M - 18.2.0 dc2bc1663786 ad189e8605bd 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.quxdec smithi190 running (2m) 22s ago 2m 16.3M - 18.2.0 dc2bc1663786 869fb60edc82 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.dolfed smithi153 *:9283,8765,8443 running (9m) 5s ago 9m 433M - 18.2.0 dc2bc1663786 0562ae98d3da 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi190.qauohf smithi190 *:8443,9283,8765 running (26s) 22s ago 6m 51.1M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 00df65339e6f 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (9m) 5s ago 9m 49.2M 2048M 18.2.0 dc2bc1663786 44f2993c90f4 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi190 smithi190 running (6m) 22s ago 6m 35.0M 2048M 18.2.0 dc2bc1663786 88aeab4dfa5e 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 5s ago 7m 15.2M - 1.5.0 0da6a335fe13 5c569e0ae0c9 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi190 smithi190 *:9100 running (6m) 22s ago 6m 17.5M - 1.5.0 0da6a335fe13 d1be4aed7b03 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 5s ago 5m 72.4M 1216M 18.2.0 dc2bc1663786 6c202dfae018 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 5s ago 5m 78.2M 1216M 18.2.0 dc2bc1663786 78ff3749996c 2024-05-06T14:31:36.636 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (5m) 5s ago 5m 68.0M 1216M 18.2.0 dc2bc1663786 3fbd486b6ac6 2024-05-06T14:31:36.637 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi190 running (4m) 22s ago 4m 49.5M 4970M 18.2.0 dc2bc1663786 79af272339eb 2024-05-06T14:31:36.637 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi190 running (4m) 22s ago 4m 45.6M 4970M 18.2.0 dc2bc1663786 6db95b0197f1 2024-05-06T14:31:36.637 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi190 running (3m) 22s ago 3m 44.2M 4970M 18.2.0 dc2bc1663786 ae3651b2db8d 2024-05-06T14:31:36.637 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (6m) 5s ago 7m 67.6M - 2.43.0 a07b618ecd1d c8f2e7c500cf 2024-05-06T14:31:36.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f27b406d3b0 msgr2=0x7f27b406f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f27b406d3b0 0x7f27b406f860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f27b8006010 tx=0x7f27b800b560 comp rx=0 tx=0).stop 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 msgr2=0x7f27c8100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 0x7f27c8100570 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f27c400d8d0 tx=0x7f27c400dbe0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 shutdown_connections 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f27b406d3b0 0x7f27b406f860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27c8100c10 0x7f27c8100570 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 --2- 172.21.15.153:0/1889240469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c8103530 0x7f27c80febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 >> 172.21.15.153:0/1889240469 conn(0x7f27c80fa630 msgr2=0x7f27c80fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 shutdown_connections 2024-05-06T14:31:36.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.651+0000 7f27d0011700 1 -- 172.21.15.153:0/1889240469 wait complete. 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 -- 172.21.15.153:0/1564477720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 msgr2=0x7fbb0c0fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 --2- 172.21.15.153:0/1564477720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c0fec50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009910 tx=0x7fbafc009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 -- 172.21.15.153:0/1564477720 shutdown_connections 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 --2- 172.21.15.153:0/1564477720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb0c0ffa40 0x7fbb0c0ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 --2- 172.21.15.153:0/1564477720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c0fec50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 -- 172.21.15.153:0/1564477720 >> 172.21.15.153:0/1564477720 conn(0x7fbb0c0f9dd0 msgr2=0x7fbb0c0fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 -- 172.21.15.153:0/1564477720 shutdown_connections 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 -- 172.21.15.153:0/1564477720 wait complete. 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 Processor -- start 2024-05-06T14:31:36.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.790+0000 7fbb12324700 1 -- start start 2024-05-06T14:31:36.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb12324700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:36.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb12324700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb0c0ffa40 0x7fbb0c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:36.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb12324700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c101360 con 0x7fbb0c0fe840 2024-05-06T14:31:36.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb12324700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c1014a0 con 0x7fbb0c0ffa40 2024-05-06T14:31:36.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:36.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c1027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53522/0 (socket says 172.21.15.153:53522) 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 -- 172.21.15.153:0/3002855861 learned_addr learned my addr 172.21.15.153:0/3002855861 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0b7fe700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb0c0ffa40 0x7fbb0c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 -- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb0c0ffa40 msgr2=0x7fbb0c100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb0c0ffa40 0x7fbb0c100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 -- 172.21.15.153:0/3002855861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0095f0 con 0x7fbb0c0fe840 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.791+0000 7fbb0bfff700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c1027d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00bda0 tx=0x7fbafc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.792+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbafc01d070 con 0x7fbb0c0fe840 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.792+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbafc00f460 con 0x7fbb0c0fe840 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.792+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c101720 con 0x7fbb0c0fe840 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.792+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbafc017630 con 0x7fbb0c0fe840 2024-05-06T14:31:36.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.792+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c101c10 con 0x7fbb0c0fe840 2024-05-06T14:31:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.793+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0c106b00 con 0x7fbb0c0fe840 2024-05-06T14:31:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.794+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7fbafc00fad0 con 0x7fbb0c0fe840 2024-05-06T14:31:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.794+0000 7fbb097fa700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fbaf406d3c0 0x7fbaf406f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.794+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fbafc08e1c0 con 0x7fbb0c0fe840 2024-05-06T14:31:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.794+0000 7fbb0b7fe700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fbaf406d3c0 0x7fbaf406f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:36.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.795+0000 7fbb0b7fe700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fbaf406d3c0 0x7fbaf406f870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb00005fd0 tx=0x7fbb00005f60 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:36.799+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7fbafc05b430 con 0x7fbb0c0fe840 2024-05-06T14:31:37.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.226+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbb0c19df40 con 0x7fbb0c0fe840 2024-05-06T14:31:37.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.227+0000 7fbb097fa700 1 -- 172.21.15.153:0/3002855861 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fbafc05afc0 con 0x7fbb0c0fe840 2024-05-06T14:31:37.213 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:31:37.213 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-05-06T14:31:37.213 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-05-06T14:31:37.214 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 13, 2024-05-06T14:31:37.215 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T14:31:37.215 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-05-06T14:31:37.215 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:31:37.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fbaf406d3c0 msgr2=0x7fbaf406f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:37.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fbaf406d3c0 0x7fbaf406f870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb00005fd0 tx=0x7fbb00005f60 comp rx=0 tx=0).stop 2024-05-06T14:31:37.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 msgr2=0x7fbb0c1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:37.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c1027d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00bda0 tx=0x7fbafc00be80 comp rx=0 tx=0).stop 2024-05-06T14:31:37.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 shutdown_connections 2024-05-06T14:31:37.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fbaf406d3c0 0x7fbaf406f870 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb0c0fe840 0x7fbb0c1027d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 --2- 172.21.15.153:0/3002855861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb0c0ffa40 0x7fbb0c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.229+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 >> 172.21.15.153:0/3002855861 conn(0x7fbb0c0f9dd0 msgr2=0x7fbb0c0fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:37.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.230+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 shutdown_connections 2024-05-06T14:31:37.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.230+0000 7fbb12324700 1 -- 172.21.15.153:0/3002855861 wait complete. 2024-05-06T14:31:37.333 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:37 smithi153 ceph-mon[27177]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:37.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:37 smithi153 ceph-mon[27177]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:37.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:37 smithi153 ceph-mon[27177]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:37.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 -- 172.21.15.153:0/2391855068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 msgr2=0x7fe75c0ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:37.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 --2- 172.21.15.153:0/2391855068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c0ff460 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe758009910 tx=0x7fe758009c20 comp rx=0 tx=0).stop 2024-05-06T14:31:37.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 -- 172.21.15.153:0/2391855068 shutdown_connections 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 --2- 172.21.15.153:0/2391855068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c100250 0x7fe75c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 --2- 172.21.15.153:0/2391855068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c0ff460 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 -- 172.21.15.153:0/2391855068 >> 172.21.15.153:0/2391855068 conn(0x7fe75c0fa5e0 msgr2=0x7fe75c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 -- 172.21.15.153:0/2391855068 shutdown_connections 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.347+0000 7fe763a61700 1 -- 172.21.15.153:0/2391855068 wait complete. 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe763a61700 1 Processor -- start 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe763a61700 1 -- start start 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe763a61700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe763a61700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c100250 0x7fe75c195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe763a61700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c196500 con 0x7fe75c0ff050 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe763a61700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c196640 con 0x7fe75c100250 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe7617fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe7617fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c1959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53542/0 (socket says 172.21.15.153:53542) 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe7617fd700 1 -- 172.21.15.153:0/2170517245 learned_addr learned my addr 172.21.15.153:0/2170517245 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe760ffc700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c100250 0x7fe75c195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe7617fd700 1 -- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c100250 msgr2=0x7fe75c195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.348+0000 7fe7617fd700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c100250 0x7fe75c195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe7617fd700 1 -- 172.21.15.153:0/2170517245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580095f0 con 0x7fe75c0ff050 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe7617fd700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c1959a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe75800bda0 tx=0x7fe75800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe75801d070 con 0x7fe75c0ff050 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c19b090 con 0x7fe75c0ff050 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c19b580 con 0x7fe75c0ff050 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe75800f460 con 0x7fe75c0ff050 2024-05-06T14:31:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.349+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe758021620 con 0x7fe75c0ff050 2024-05-06T14:31:37.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.350+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c107ba0 con 0x7fe75c0ff050 2024-05-06T14:31:37.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.351+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7fe75800f5d0 con 0x7fe75c0ff050 2024-05-06T14:31:37.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.351+0000 7fe7527fc700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fe74806d3c0 0x7fe74806f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:37.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.351+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fe75808dfe0 con 0x7fe75c0ff050 2024-05-06T14:31:37.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.351+0000 7fe760ffc700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fe74806d3c0 0x7fe74806f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:37.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.352+0000 7fe760ffc700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fe74806d3c0 0x7fe74806f870 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe74c005950 tx=0x7fe74c00b410 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:37.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.360+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7fe75805b250 con 0x7fe75c0ff050 2024-05-06T14:31:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:37 smithi190 ceph-mon[33671]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:37 smithi190 ceph-mon[33671]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:37.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:37 smithi190 ceph-mon[33671]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:37.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.659+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe75c068ae0 con 0x7fe74806d3c0 2024-05-06T14:31:37.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.661+0000 7fe7527fc700 1 -- 172.21.15.153:0/2170517245 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fe75c068ae0 con 0x7fe74806d3c0 2024-05-06T14:31:37.647 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:31:37.647 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb", 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "1/2 daemons upgraded", 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "", 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-05-06T14:31:37.648 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:31:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fe74806d3c0 msgr2=0x7fe74806f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fe74806d3c0 0x7fe74806f870 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe74c005950 tx=0x7fe74c00b410 comp rx=0 tx=0).stop 2024-05-06T14:31:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 msgr2=0x7fe75c1959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c1959a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe75800bda0 tx=0x7fe75800bdd0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 shutdown_connections 2024-05-06T14:31:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fe74806d3c0 0x7fe74806f870 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c0ff050 0x7fe75c1959a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 --2- 172.21.15.153:0/2170517245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c100250 0x7fe75c195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.663+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 >> 172.21.15.153:0/2170517245 conn(0x7fe75c0fa5e0 msgr2=0x7fe75c0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.664+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 shutdown_connections 2024-05-06T14:31:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:37.664+0000 7fe763a61700 1 -- 172.21.15.153:0/2170517245 wait complete. 2024-05-06T14:31:38.391 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:38 smithi190 ceph-mon[33671]: fsmap cephfs:2 {0=cephfs.smithi153.ftbnnk=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:31:38.391 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:38 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3002855861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:31:38.391 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:38 smithi190 ceph-mon[33671]: pgmap v12: 65 pgs: 65 active+clean; 68 MiB data, 402 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 386 op/s 2024-05-06T14:31:38.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:38 smithi153 ceph-mon[27177]: fsmap cephfs:2 {0=cephfs.smithi153.ftbnnk=up:active,1=cephfs.smithi190.psnzbz=up:active} 2 up:standby 2024-05-06T14:31:38.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:38 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3002855861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:31:38.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:38 smithi153 ceph-mon[27177]: pgmap v12: 65 pgs: 65 active+clean; 68 MiB data, 402 MiB used, 536 GiB / 536 GiB avail; 17 KiB/s rd, 5.8 MiB/s wr, 386 op/s 2024-05-06T14:31:39.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:39 smithi190 ceph-mon[33671]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:39.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:39 smithi153 ceph-mon[27177]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:31:40.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:40 smithi190 ceph-mon[33671]: pgmap v13: 65 pgs: 65 active+clean; 68 MiB data, 402 MiB used, 536 GiB / 536 GiB avail; 4.9 MiB/s wr, 351 op/s 2024-05-06T14:31:40.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:40 smithi153 ceph-mon[27177]: pgmap v13: 65 pgs: 65 active+clean; 68 MiB data, 402 MiB used, 536 GiB / 536 GiB avail; 4.9 MiB/s wr, 351 op/s 2024-05-06T14:31:42.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:42 smithi190 ceph-mon[33671]: pgmap v14: 65 pgs: 65 active+clean; 110 MiB data, 582 MiB used, 536 GiB / 536 GiB avail; 8.8 MiB/s wr, 847 op/s 2024-05-06T14:31:42.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:42 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:42.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:42 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:42.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:42 smithi153 ceph-mon[27177]: pgmap v14: 65 pgs: 65 active+clean; 110 MiB data, 582 MiB used, 536 GiB / 536 GiB avail; 8.8 MiB/s wr, 847 op/s 2024-05-06T14:31:42.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:42 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:42.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:42 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:44.673 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:44 smithi190 ceph-mon[33671]: pgmap v15: 65 pgs: 65 active+clean; 110 MiB data, 582 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 618 op/s 2024-05-06T14:31:44.674 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:44 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:44.674 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:44 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:44.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:44 smithi153 ceph-mon[27177]: pgmap v15: 65 pgs: 65 active+clean; 110 MiB data, 582 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 618 op/s 2024-05-06T14:31:44.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:44 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:44.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:44 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:46.103 INFO:tasks.workunit.client.0.smithi153.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-05-06T14:31:46.104 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-05-06T14:31:46.104 INFO:tasks.workunit.client.0.smithi153.stderr:+ make 2024-05-06T14:31:46.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:46 smithi190 ceph-mon[33671]: pgmap v16: 65 pgs: 65 active+clean; 110 MiB data, 582 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 618 op/s 2024-05-06T14:31:46.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:46 smithi153 ceph-mon[27177]: pgmap v16: 65 pgs: 65 active+clean; 110 MiB data, 582 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 618 op/s 2024-05-06T14:31:47.662 INFO:tasks.workunit.client.0.smithi153.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-05-06T14:31:48.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: pgmap v17: 65 pgs: 65 active+clean; 155 MiB data, 768 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s wr, 975 op/s 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: Adjusting osd_memory_target on smithi190 to 2240M 2024-05-06T14:31:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:48.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:31:48.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:31:48.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: Updating smithi153:/etc/ceph/ceph.conf 2024-05-06T14:31:48.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:48 smithi153 ceph-mon[27177]: Updating smithi190:/etc/ceph/ceph.conf 2024-05-06T14:31:48.857 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: pgmap v17: 65 pgs: 65 active+clean; 155 MiB data, 768 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s wr, 975 op/s 2024-05-06T14:31:48.857 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:48.857 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: Adjusting osd_memory_target on smithi190 to 2240M 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: Updating smithi153:/etc/ceph/ceph.conf 2024-05-06T14:31:48.858 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:48 smithi190 ceph-mon[33671]: Updating smithi190:/etc/ceph/ceph.conf 2024-05-06T14:31:49.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.937 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.938 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:49 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: Updating smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: Updating smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T14:31:49.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:49.942 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:49 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:31:50.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:50 smithi190 ceph-mon[33671]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-05-06T14:31:50.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:50 smithi190 ceph-mon[33671]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-05-06T14:31:50.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:50 smithi190 ceph-mon[33671]: pgmap v18: 65 pgs: 65 active+clean; 155 MiB data, 768 MiB used, 536 GiB / 536 GiB avail; 7.8 MiB/s wr, 853 op/s 2024-05-06T14:31:50.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:50 smithi153 ceph-mon[27177]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-05-06T14:31:50.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:50 smithi153 ceph-mon[27177]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-05-06T14:31:50.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:50 smithi153 ceph-mon[27177]: pgmap v18: 65 pgs: 65 active+clean; 155 MiB data, 768 MiB used, 536 GiB / 536 GiB avail; 7.8 MiB/s wr, 853 op/s 2024-05-06T14:31:52.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:52 smithi190 ceph-mon[33671]: pgmap v19: 65 pgs: 65 active+clean; 171 MiB data, 816 MiB used, 536 GiB / 536 GiB avail; 9.1 MiB/s wr, 874 op/s 2024-05-06T14:31:53.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:52 smithi153 ceph-mon[27177]: pgmap v19: 65 pgs: 65 active+clean; 171 MiB data, 816 MiB used, 536 GiB / 536 GiB avail; 9.1 MiB/s wr, 874 op/s 2024-05-06T14:31:53.032 INFO:tasks.workunit.client.0.smithi153.stderr:++ readlink -f fsstress 2024-05-06T14:31:53.034 INFO:tasks.workunit.client.0.smithi153.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-05-06T14:31:53.034 INFO:tasks.workunit.client.0.smithi153.stderr:+ popd 2024-05-06T14:31:53.034 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-05-06T14:31:53.034 INFO:tasks.workunit.client.0.smithi153.stderr:+ popd 2024-05-06T14:31:53.035 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp 2024-05-06T14:31:53.035 INFO:tasks.workunit.client.0.smithi153.stderr:++ mktemp -d -p . 2024-05-06T14:31:53.036 INFO:tasks.workunit.client.0.smithi153.stderr:+ T=./tmp.gLjgz7ghau 2024-05-06T14:31:53.036 INFO:tasks.workunit.client.0.smithi153.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.gLjgz7ghau -l 1 -n 1000 -p 10 -v 2024-05-06T14:31:53.041 INFO:tasks.workunit.client.0.smithi153.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh: line 16: /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress: cannot execute binary file: Exec format error 2024-05-06T14:31:53.042 DEBUG:teuthology.orchestra.run:got remote process result: 126 2024-05-06T14:31:53.042 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-05-06T14:31:53.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-06T14:31:53.174 INFO:tasks.workunit.client.1.smithi190.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-05-06T14:31:53.174 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-05-06T14:31:53.175 INFO:tasks.workunit.client.1.smithi190.stderr:+ make 2024-05-06T14:31:53.264 INFO:tasks.workunit.client.1.smithi190.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-05-06T14:31:53.409 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi153 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=70ce9539b62a15b9de429c1f0e91a1e012b099aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2024-05-06T14:31:53.639 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6fcda01840dc42c09fa3d59355acdcff Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi153 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=70ce9539b62a15b9de429c1f0e91a1e012b099aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2024-05-06T14:31:53.642 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-06T14:31:53.654 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-06T14:31:53.654 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:31:53.654 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-06T14:31:53.660 INFO:tasks.workunit.client.1.smithi190.stderr:++ readlink -f fsstress 2024-05-06T14:31:53.660 INFO:tasks.workunit.client.1.smithi190.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-05-06T14:31:53.660 INFO:tasks.workunit.client.1.smithi190.stderr:+ popd 2024-05-06T14:31:53.661 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-05-06T14:31:53.661 INFO:tasks.workunit.client.1.smithi190.stderr:+ popd 2024-05-06T14:31:53.661 INFO:tasks.workunit.client.1.smithi190.stdout:~/cephtest/mnt.1/client.1/tmp 2024-05-06T14:31:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 126 2024-05-06T14:31:53.663 INFO:tasks.workunit.client.1.smithi190.stderr:++ mktemp -d -p . 2024-05-06T14:31:53.663 INFO:tasks.workunit.client.1.smithi190.stderr:+ T=./tmp.bEjBSYOw50 2024-05-06T14:31:53.663 INFO:tasks.workunit.client.1.smithi190.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.bEjBSYOw50 -l 1 -n 1000 -p 10 -v 2024-05-06T14:31:53.663 INFO:tasks.workunit.client.1.smithi190.stderr:/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh: line 16: /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress: cannot execute binary file: Exec format error 2024-05-06T14:31:53.663 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-05-06T14:31:53.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-05-06T14:31:53.676 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-05-06T14:31:53.676 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-06T14:31:53.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T14:31:53.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 -- 172.21.15.153:0/14697764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 msgr2=0x7f6ac01098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:53.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/14697764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01098a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40098b0 tx=0x7f6ab402bcd0 comp rx=0 tx=0).stop 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 -- 172.21.15.153:0/14697764 shutdown_connections 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/14697764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01098a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/14697764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ac0100aa0 0x7f6ac0100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 -- 172.21.15.153:0/14697764 >> 172.21.15.153:0/14697764 conn(0x7f6ac00fa230 msgr2=0x7f6ac00fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 -- 172.21.15.153:0/14697764 shutdown_connections 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 -- 172.21.15.153:0/14697764 wait complete. 2024-05-06T14:31:53.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.885+0000 7f6ac5b99640 1 Processor -- start 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6ac5b99640 1 -- start start 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6ac5b99640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ac0100aa0 0x7f6ac0195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6ac5b99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6ac5b99640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac0196030 con 0x7f6ac01013b0 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6ac5b99640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac0199e00 con 0x7f6ac0100aa0 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40472/0 (socket says 172.21.15.153:40472) 2024-05-06T14:31:53.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 -- 172.21.15.153:0/2462548154 learned_addr learned my addr 172.21.15.153:0/2462548154 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ac0100aa0 0x7f6ac0195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 -- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ac0100aa0 msgr2=0x7f6ac0195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ac0100aa0 0x7f6ac0195460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 -- 172.21.15.153:0/2462548154 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab4009560 con 0x7f6ac01013b0 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abeffd640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01959a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009880 tx=0x7f6ab4004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.886+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ab4004420 con 0x7f6ac01013b0 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.887+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ab40054b0 con 0x7f6ac01013b0 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.887+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac019a080 con 0x7f6ac01013b0 2024-05-06T14:31:53.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.887+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ab4003db0 con 0x7f6ac01013b0 2024-05-06T14:31:53.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.887+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac019a580 con 0x7f6ac01013b0 2024-05-06T14:31:53.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.888+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac004e000 con 0x7f6ac01013b0 2024-05-06T14:31:53.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.888+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f6ab40035f0 con 0x7f6ac01013b0 2024-05-06T14:31:53.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.888+0000 7f6abcff9640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6aa806d3d0 0x7f6aa806f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:53.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.888+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6ab40aeeb0 con 0x7f6ac01013b0 2024-05-06T14:31:53.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.889+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6aa806d3d0 0x7f6aa806f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:53.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.889+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6aa806d3d0 0x7f6aa806f880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009bd0 tx=0x7f6ab0005f90 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:53.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:53.893+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f6ab40787b0 con 0x7f6ac01013b0 2024-05-06T14:31:54.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.179+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ac0066860 con 0x7f6ac01013b0 2024-05-06T14:31:54.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.181+0000 7f6abcff9640 1 -- 172.21.15.153:0/2462548154 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+15543 (secure 0 0 0) 0x7f6ab407be70 con 0x7f6ac01013b0 2024-05-06T14:31:54.168 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:31:54.168 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":49,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","created":"2024-05-06T14:22:12.905455+0000","modified":"2024-05-06T14:31:19.297414+0000","last_up_change":"2024-05-06T14:27:54.584488+0000","last_in_change":"2024-05-06T14:27:35.267866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T14:26:41.240977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-05-06T14:29:08.798039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-05-06T14:29:09.557951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de83eb29-8176-4eb5-9be6-64b9e5449892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6803","nonce":908509767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6805","nonce":908509767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6809","nonce":908509767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6807","nonce":908509767}]},"public_addr":"172.21.15.153:6803/908509767","cluster_addr":"172.21.15.153:6805/908509767","heartbeat_back_addr":"172.21.15.153:6809/908509767","heartbeat_front_addr":"172.21.15.153:6807/908509767","state":["exists","up"]},{"osd":1,"uuid":"24a1c7ac-a168-4e7d-8c34-3e4c73408756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6811","nonce":3716718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6813","nonce":3716718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6817","nonce":3716718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6815","nonce":3716718209}]},"public_addr":"172.21.15.153:6811/3716718209","cluster_addr":"172.21.15.153:6813/3716718209","heartbeat_back_addr":"172.21.15.153:6817/3716718209","heartbeat_front_addr":"172.21.15.153:6815/3716718209","state":["exists","up"]},{"osd":2,"uuid":"5b89edb2-1c0e-4481-91ed-157d767c0c05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6819","nonce":3169611803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6821","nonce":3169611803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6825","nonce":3169611803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6823","nonce":3169611803}]},"public_addr":"172.21.15.153:6819/3169611803","cluster_addr":"172.21.15.153:6821/3169611803","heartbeat_back_addr":"172.21.15.153:6825/3169611803","heartbeat_front_addr":"172.21.15.153:6823/3169611803","state":["exists","up"]},{"osd":3,"uuid":"74b7823e-2b78-4570-9b7a-89abc9d7fbb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6801","nonce":492034660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6803","nonce":492034660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6807","nonce":492034660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6805","nonce":492034660}]},"public_addr":"172.21.15.190:6801/492034660","cluster_addr":"172.21.15.190:6803/492034660","heartbeat_back_addr":"172.21.15.190:6807/492034660","heartbeat_front_addr":"172.21.15.190:6805/492034660","state":["exists","up"]},{"osd":4,"uuid":"01290c44-3bf1-4da3-bac3-4dff2453cedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6809","nonce":1342651346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6811","nonce":1342651346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6815","nonce":1342651346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6813","nonce":1342651346}]},"public_addr":"172.21.15.190:6809/1342651346","cluster_addr":"172.21.15.190:6811/1342651346","heartbeat_back_addr":"172.21.15.190:6815/1342651346","heartbeat_front_addr":"172.21.15.190:6813/1342651346","state":["exists","up"]},{"osd":5,"uuid":"be366c2e-63d4-4496-b401-772dbe1a2444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6817","nonce":452463735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6819","nonce":452463735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6823","nonce":452463735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6821","nonce":452463735}]},"public_addr":"172.21.15.190:6817/452463735","cluster_addr":"172.21.15.190:6819/452463735","heartbeat_back_addr":"172.21.15.190:6823/452463735","heartbeat_front_addr":"172.21.15.190:6821/452463735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:25:42.510404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:11.276096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:37.855142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:01.189580+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:26.756150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:52.720929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6801/1473246849":"2024-05-07T14:31:18.651849+0000","172.21.15.153:6800/1473246849":"2024-05-07T14:31:18.651849+0000","172.21.15.153:0/3340397168":"2024-05-07T14:31:18.651849+0000","172.21.15.153:0/1418596404":"2024-05-07T14:31:18.651849+0000","172.21.15.153:0/3898448400":"2024-05-07T14:31:18.651849+0000","172.21.15.153:6827/4248251632":"2024-05-07T14:29:26.017160+0000","172.21.15.153:0/3158057303":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6826/4248251632":"2024-05-07T14:29:26.017160+0000","172.21.15.153:6801/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3536681135":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1199411437":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6800/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1309940234":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2948592992":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/4101428691":"2024-05-07T14:31:18.651849+0000","172.21.15.153:6801/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/1311303452":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2161963253":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6801/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1460873727":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2433042994":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6800/2228103355":"2024-05-07T14:24:24.923607+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T14:31:54.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6aa806d3d0 msgr2=0x7f6aa806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6aa806d3d0 0x7f6aa806f880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009bd0 tx=0x7f6ab0005f90 comp rx=0 tx=0).stop 2024-05-06T14:31:54.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 msgr2=0x7f6ac01959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01959a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009880 tx=0x7f6ab4004940 comp rx=0 tx=0).stop 2024-05-06T14:31:54.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 shutdown_connections 2024-05-06T14:31:54.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6aa806d3d0 0x7f6aa806f880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac01013b0 0x7f6ac01959a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 --2- 172.21.15.153:0/2462548154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ac0100aa0 0x7f6ac0195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 >> 172.21.15.153:0/2462548154 conn(0x7f6ac00fa230 msgr2=0x7f6ac01040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:54.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 shutdown_connections 2024-05-06T14:31:54.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.185+0000 7f6ac5b99640 1 -- 172.21.15.153:0/2462548154 wait complete. 2024-05-06T14:31:54.192 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T14:26:41.240977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-05-06T14:29:08.798039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-05-06T14:29:09.557951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-06T14:31:54.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-06T14:31:54.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.320+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3932013020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd810ddc0 msgr2=0x7f2dd810e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.320+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3932013020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd810ddc0 0x7f2dd810e190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009c20 tx=0x7f2dd002c1d0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.321+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3932013020 shutdown_connections 2024-05-06T14:31:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.321+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3932013020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd8070e60 0x7f2dd80712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.321+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3932013020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd810ddc0 0x7f2dd810e190 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009c20 tx=0x7f2dd002c1d0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.321+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3932013020 >> 172.21.15.153:0/3932013020 conn(0x7f2dd806b8d0 msgr2=0x7f2dd806bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.325+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3932013020 shutdown_connections 2024-05-06T14:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.326+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3932013020 wait complete. 2024-05-06T14:31:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.327+0000 7f2ddd07d640 1 Processor -- start 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.328+0000 7f2ddd07d640 1 -- start start 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2ddd07d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd8070e60 0x7f2dd81a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2ddd07d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 0x7f2dd81a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2ddd07d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd81a4de0 con 0x7f2dd8070e60 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2ddd07d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd81a4f50 con 0x7f2dd81a47f0 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2dd659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 0x7f2dd81a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2dd659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 0x7f2dd81a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:40650/0 (socket says 172.21.15.153:40650) 2024-05-06T14:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.329+0000 7f2dd659c640 1 -- 172.21.15.153:0/3356363139 learned_addr learned my addr 172.21.15.153:0/3356363139 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.330+0000 7f2dd6d9d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd8070e60 0x7f2dd81a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.330+0000 7f2dd6d9d640 1 -- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 msgr2=0x7f2dd81a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.330+0000 7f2dd6d9d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 0x7f2dd81a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.330+0000 7f2dd6d9d640 1 -- 172.21.15.153:0/3356363139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0009900 con 0x7f2dd8070e60 2024-05-06T14:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.330+0000 7f2dd659c640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 0x7f2dd81a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-06T14:31:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.330+0000 7f2dd6d9d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd8070e60 0x7f2dd81a42b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0008000 tx=0x7f2dd0032b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.333+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dd0034820 con 0x7f2dd8070e60 2024-05-06T14:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.334+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2dd0003e20 con 0x7f2dd8070e60 2024-05-06T14:31:54.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.334+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dd0004840 con 0x7f2dd8070e60 2024-05-06T14:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.334+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd81a91f0 con 0x7f2dd8070e60 2024-05-06T14:31:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.334+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd81a9690 con 0x7f2dd8070e60 2024-05-06T14:31:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.338+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f2dd003b040 con 0x7f2dd8070e60 2024-05-06T14:31:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.338+0000 7f2dbffff640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2dc006d430 0x7f2dc006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.339+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f2dd00af6c0 con 0x7f2dd8070e60 2024-05-06T14:31:54.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.339+0000 7f2dd659c640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2dc006d430 0x7f2dc006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:54.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.347+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc4005280 con 0x7f2dd8070e60 2024-05-06T14:31:54.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.351+0000 7f2dd659c640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2dc006d430 0x7f2dc006f8e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0058c0 tx=0x7f2dcc00a380 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.360+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f2dd0078f70 con 0x7f2dd8070e60 2024-05-06T14:31:54.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.621+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2dc4005ab0 con 0x7f2dd8070e60 2024-05-06T14:31:54.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.621+0000 7f2dbffff640 1 -- 172.21.15.153:0/3356363139 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2dd00390a0 con 0x7f2dd8070e60 2024-05-06T14:31:54.608 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-05-06T14:31:54.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2dc006d430 msgr2=0x7f2dc006f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2dc006d430 0x7f2dc006f8e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0058c0 tx=0x7f2dcc00a380 comp rx=0 tx=0).stop 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd8070e60 msgr2=0x7f2dd81a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd8070e60 0x7f2dd81a42b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0008000 tx=0x7f2dd0032b10 comp rx=0 tx=0).stop 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 shutdown_connections 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2dc006d430 0x7f2dc006f8e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2dd81a47f0 0x7f2dd81a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 --2- 172.21.15.153:0/3356363139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd8070e60 0x7f2dd81a42b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.623+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 >> 172.21.15.153:0/3356363139 conn(0x7f2dd806b8d0 msgr2=0x7f2dd806ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:54.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.624+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 shutdown_connections 2024-05-06T14:31:54.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.624+0000 7f2ddd07d640 1 -- 172.21.15.153:0/3356363139 wait complete. 2024-05-06T14:31:54.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-05-06T14:31:54.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 -- 172.21.15.153:0/993014527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 msgr2=0x7f8964105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 --2- 172.21.15.153:0/993014527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964105a50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8958009880 tx=0x7f895802bce0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 -- 172.21.15.153:0/993014527 shutdown_connections 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 --2- 172.21.15.153:0/993014527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89640ff670 0x7f89640ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 --2- 172.21.15.153:0/993014527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964105a50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 -- 172.21.15.153:0/993014527 >> 172.21.15.153:0/993014527 conn(0x7f89640fb190 msgr2=0x7f89640fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 -- 172.21.15.153:0/993014527 shutdown_connections 2024-05-06T14:31:54.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.751+0000 7f896b036640 1 -- 172.21.15.153:0/993014527 wait complete. 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f896b036640 1 Processor -- start 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f896b036640 1 -- start start 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f896b036640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89640ff670 0x7f89641952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f896b036640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f896b036640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8964195e20 con 0x7f8964105680 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f896b036640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8964195f90 con 0x7f89640ff670 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f8963fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f8963fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40530/0 (socket says 172.21.15.153:40530) 2024-05-06T14:31:54.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.752+0000 7f8963fff640 1 -- 172.21.15.153:0/136018390 learned_addr learned my addr 172.21.15.153:0/136018390 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8963fff640 1 -- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89640ff670 msgr2=0x7f89641952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8963fff640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89640ff670 0x7f89641952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8963fff640 1 -- 172.21.15.153:0/136018390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8958009560 con 0x7f8964105680 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8963fff640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964195820 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f895000c8a0 tx=0x7f895000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8950007b80 con 0x7f8964105680 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8950007ce0 con 0x7f8964105680 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8964199e20 con 0x7f8964105680 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8950018810 con 0x7f8964105680 2024-05-06T14:31:54.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.753+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896419a380 con 0x7f8964105680 2024-05-06T14:31:54.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.754+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896404e000 con 0x7f8964105680 2024-05-06T14:31:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.755+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f8950018a80 con 0x7f8964105680 2024-05-06T14:31:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.755+0000 7f8961ffb640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f895406d330 0x7f895406f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.755+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f895008cfd0 con 0x7f8964105680 2024-05-06T14:31:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.755+0000 7f8968dd2640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f895406d330 0x7f895406f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:54.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.756+0000 7f8968dd2640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f895406d330 0x7f895406f7e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8958036040 tx=0x7f8958000bc0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:54.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.759+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f8950056670 con 0x7f8964105680 2024-05-06T14:31:54.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:54 smithi153 ceph-mon[27177]: pgmap v20: 65 pgs: 65 active+clean; 171 MiB data, 816 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 378 op/s 2024-05-06T14:31:54.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:54 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2462548154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:31:54.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:54 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3356363139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:31:54.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:54 smithi190 ceph-mon[33671]: pgmap v20: 65 pgs: 65 active+clean; 171 MiB data, 816 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 378 op/s 2024-05-06T14:31:54.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:54 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2462548154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:31:54.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:54 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3356363139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:31:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.957+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f8964196630 con 0x7f8964105680 2024-05-06T14:31:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.957+0000 7f8961ffb640 1 -- 172.21.15.153:0/136018390 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v49) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8950059d30 con 0x7f8964105680 2024-05-06T14:31:54.944 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-05-06T14:31:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f895406d330 msgr2=0x7f895406f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f895406d330 0x7f895406f7e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8958036040 tx=0x7f8958000bc0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 msgr2=0x7f8964195820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:54.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964195820 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f895000c8a0 tx=0x7f895000cd60 comp rx=0 tx=0).stop 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 shutdown_connections 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f895406d330 0x7f895406f7e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964105680 0x7f8964195820 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 --2- 172.21.15.153:0/136018390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89640ff670 0x7f89641952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 >> 172.21.15.153:0/136018390 conn(0x7f89640fb190 msgr2=0x7f89640fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 shutdown_connections 2024-05-06T14:31:54.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:54.959+0000 7f896b036640 1 -- 172.21.15.153:0/136018390 wait complete. 2024-05-06T14:31:54.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-05-06T14:31:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 -- 172.21.15.153:0/4152487454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 msgr2=0x7f9bd40ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 --2- 172.21.15.153:0/4152487454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd40ffac0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009880 tx=0x7f9bc802bce0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 -- 172.21.15.153:0/4152487454 shutdown_connections 2024-05-06T14:31:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 --2- 172.21.15.153:0/4152487454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd40ffac0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 --2- 172.21.15.153:0/4152487454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd4105660 0x7f9bd4105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 -- 172.21.15.153:0/4152487454 >> 172.21.15.153:0/4152487454 conn(0x7f9bd40fb190 msgr2=0x7f9bd40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 -- 172.21.15.153:0/4152487454 shutdown_connections 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.079+0000 7f9bda93c640 1 -- 172.21.15.153:0/4152487454 wait complete. 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.080+0000 7f9bda93c640 1 Processor -- start 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.080+0000 7f9bda93c640 1 -- start start 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.080+0000 7f9bda93c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd4195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.080+0000 7f9bda93c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd4105660 0x7f9bd4195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.080+0000 7f9bda93c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4195ef0 con 0x7f9bd40ff650 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.080+0000 7f9bda93c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4199cc0 con 0x7f9bd4105660 2024-05-06T14:31:55.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd4195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd4195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40550/0 (socket says 172.21.15.153:40550) 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 -- 172.21.15.153:0/196296832 learned_addr learned my addr 172.21.15.153:0/196296832 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd37fe640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd4105660 0x7f9bd4195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 -- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd4105660 msgr2=0x7f9bd4195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd4105660 0x7f9bd4195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 -- 172.21.15.153:0/196296832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8009560 con 0x7f9bd40ff650 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd3fff640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd4195320 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400d8d0 tx=0x7f9bc400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc400bea0 con 0x7f9bd40ff650 2024-05-06T14:31:55.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd4199fa0 con 0x7f9bd40ff650 2024-05-06T14:31:55.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9bc4009c00 con 0x7f9bd40ff650 2024-05-06T14:31:55.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd419a500 con 0x7f9bd40ff650 2024-05-06T14:31:55.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.081+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bc400f780 con 0x7f9bd40ff650 2024-05-06T14:31:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.083+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd404e000 con 0x7f9bd40ff650 2024-05-06T14:31:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.083+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f9bc400fa40 con 0x7f9bd40ff650 2024-05-06T14:31:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.083+0000 7f9bd17fa640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f9bbc06d330 0x7f9bbc06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.084+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f9bc408cfd0 con 0x7f9bd40ff650 2024-05-06T14:31:55.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.084+0000 7f9bd37fe640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f9bbc06d330 0x7f9bbc06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:55.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.084+0000 7f9bd37fe640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f9bbc06d330 0x7f9bbc06f7e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4196930 tx=0x7f9bc8000bc0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.092+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f9bc405a240 con 0x7f9bd40ff650 2024-05-06T14:31:55.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.277+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f9bd4107b30 con 0x7f9bd40ff650 2024-05-06T14:31:55.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.278+0000 7f9bd17fa640 1 -- 172.21.15.153:0/196296832 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v49) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9bc4059dd0 con 0x7f9bd40ff650 2024-05-06T14:31:55.265 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-05-06T14:31:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f9bbc06d330 msgr2=0x7f9bbc06f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f9bbc06d330 0x7f9bbc06f7e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4196930 tx=0x7f9bc8000bc0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 msgr2=0x7f9bd4195320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd4195320 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400d8d0 tx=0x7f9bc400dd90 comp rx=0 tx=0).stop 2024-05-06T14:31:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 shutdown_connections 2024-05-06T14:31:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f9bbc06d330 0x7f9bbc06f7e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bd4105660 0x7f9bd4195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 --2- 172.21.15.153:0/196296832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bd40ff650 0x7f9bd4195320 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 >> 172.21.15.153:0/196296832 conn(0x7f9bd40fb190 msgr2=0x7f9bd40fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:55.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 shutdown_connections 2024-05-06T14:31:55.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.280+0000 7f9bda93c640 1 -- 172.21.15.153:0/196296832 wait complete. 2024-05-06T14:31:55.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-06T14:31:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.397+0000 7f6ee33b9640 1 -- 172.21.15.153:0/284187011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc100850 msgr2=0x7f6edc100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.397+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/284187011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc100850 0x7f6edc100c20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80098b0 tx=0x7f6ed802bd60 comp rx=0 tx=0).stop 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 -- 172.21.15.153:0/284187011 shutdown_connections 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/284187011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6edc101160 0x7f6edc109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/284187011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc100850 0x7f6edc100c20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 -- 172.21.15.153:0/284187011 >> 172.21.15.153:0/284187011 conn(0x7f6edc075270 msgr2=0x7f6edc075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 -- 172.21.15.153:0/284187011 shutdown_connections 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 -- 172.21.15.153:0/284187011 wait complete. 2024-05-06T14:31:55.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 Processor -- start 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.398+0000 7f6ee33b9640 1 -- start start 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee33b9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6edc100850 0x7f6edc195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee33b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 0x7f6edc1957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee33b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc195e40 con 0x7f6edc101160 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee33b9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc199c10 con 0x7f6edc100850 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 0x7f6edc1957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:55.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 0x7f6edc1957b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40572/0 (socket says 172.21.15.153:40572) 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 -- 172.21.15.153:0/2646283638 learned_addr learned my addr 172.21.15.153:0/2646283638 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee1155640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6edc100850 0x7f6edc195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 -- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6edc100850 msgr2=0x7f6edc195270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6edc100850 0x7f6edc195270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 -- 172.21.15.153:0/2646283638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8009560 con 0x7f6edc101160 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee0954640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 0x7f6edc1957b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00c8f0 tx=0x7f6ecc00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ecc007b80 con 0x7f6edc101160 2024-05-06T14:31:55.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc199ef0 con 0x7f6edc101160 2024-05-06T14:31:55.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ecc007ce0 con 0x7f6edc101160 2024-05-06T14:31:55.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ecc00f450 con 0x7f6edc101160 2024-05-06T14:31:55.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.399+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc19a450 con 0x7f6edc101160 2024-05-06T14:31:55.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.400+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc04e000 con 0x7f6edc101160 2024-05-06T14:31:55.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.401+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f6ecc004f60 con 0x7f6edc101160 2024-05-06T14:31:55.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.402+0000 7f6ed27fc640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6ec806d3d0 0x7f6ec806f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:55.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.402+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6ecc08ceb0 con 0x7f6edc101160 2024-05-06T14:31:55.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.402+0000 7f6ee1155640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6ec806d3d0 0x7f6ec806f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:55.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.402+0000 7f6ee1155640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6ec806d3d0 0x7f6ec806f880 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8006010 tx=0x7f6ed8000bc0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:55.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.406+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f6ecc056710 con 0x7f6edc101160 2024-05-06T14:31:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.602+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6edc106cc0 con 0x7f6edc101160 2024-05-06T14:31:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.603+0000 7f6ed27fc640 1 -- 172.21.15.153:0/2646283638 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 23 entries v49) v1 ==== 81+0+1385 (secure 0 0 0) 0x7f6ecc01d580 con 0x7f6edc101160 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/1473246849 2024-05-07T14:31:18.651849+0000 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/1473246849 2024-05-07T14:31:18.651849+0000 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3340397168 2024-05-07T14:31:18.651849+0000 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1418596404 2024-05-07T14:31:18.651849+0000 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3898448400 2024-05-07T14:31:18.651849+0000 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6827/4248251632 2024-05-07T14:29:26.017160+0000 2024-05-06T14:31:55.590 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3158057303 2024-05-07T14:24:24.923607+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6826/4248251632 2024-05-07T14:29:26.017160+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/4102256583 2024-05-07T14:22:40.569350+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3536681135 2024-05-07T14:22:40.569350+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1199411437 2024-05-07T14:22:40.569350+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/4102256583 2024-05-07T14:22:40.569350+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1309940234 2024-05-07T14:23:12.780977+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2948592992 2024-05-07T14:23:12.780977+0000 2024-05-06T14:31:55.591 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/585874296 2024-05-07T14:23:12.780977+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/4101428691 2024-05-07T14:31:18.651849+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/585874296 2024-05-07T14:23:12.780977+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1311303452 2024-05-07T14:24:24.923607+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2161963253 2024-05-07T14:22:40.569350+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/2228103355 2024-05-07T14:24:24.923607+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1460873727 2024-05-07T14:23:12.780977+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2433042994 2024-05-07T14:24:24.923607+0000 2024-05-06T14:31:55.592 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/2228103355 2024-05-07T14:24:24.923607+0000 2024-05-06T14:31:55.593 INFO:teuthology.orchestra.run.smithi153.stderr:listed 23 entries 2024-05-06T14:31:55.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6ec806d3d0 msgr2=0x7f6ec806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6ec806d3d0 0x7f6ec806f880 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8006010 tx=0x7f6ed8000bc0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 msgr2=0x7f6edc1957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:55.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 0x7f6edc1957b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00c8f0 tx=0x7f6ecc00cdb0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 shutdown_connections 2024-05-06T14:31:55.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f6ec806d3d0 0x7f6ec806f880 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6edc101160 0x7f6edc1957b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 --2- 172.21.15.153:0/2646283638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6edc100850 0x7f6edc195270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:55.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 >> 172.21.15.153:0/2646283638 conn(0x7f6edc075270 msgr2=0x7f6edc103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:55.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 shutdown_connections 2024-05-06T14:31:55.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:55.605+0000 7f6ee33b9640 1 -- 172.21.15.153:0/2646283638 wait complete. 2024-05-06T14:31:55.603 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-06T14:31:55.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:31:55.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-06T14:31:55.629 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:31:55.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:31:55.630 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-06T14:31:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:55 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/136018390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-06T14:31:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:55 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/196296832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-06T14:31:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:55 smithi153 ceph-mon[27177]: pgmap v21: 65 pgs: 65 active+clean; 171 MiB data, 816 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 378 op/s 2024-05-06T14:31:55.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:55 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2646283638' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:31:55.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:55 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/136018390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-06T14:31:55.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:55 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/196296832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-06T14:31:55.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:55 smithi190 ceph-mon[33671]: pgmap v21: 65 pgs: 65 active+clean; 171 MiB data, 816 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s wr, 378 op/s 2024-05-06T14:31:55.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:55 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2646283638' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:31:58.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:58 smithi153 ceph-mon[27177]: pgmap v22: 65 pgs: 65 active+clean; 193 MiB data, 901 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s wr, 541 op/s 2024-05-06T14:31:58.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:58 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:58.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:58 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:58.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:58 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:31:58.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:58 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T14:31:58.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:58 smithi190 ceph-mon[33671]: pgmap v22: 65 pgs: 65 active+clean; 193 MiB data, 901 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s wr, 541 op/s 2024-05-06T14:31:58.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:58 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:58.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:58 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:58.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:58 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:31:58.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:58 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T14:31:59.256 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:31:59.257 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-06T14:31:59.274 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-05-06T14:31:59.275 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-06T14:31:59.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-06T14:31:59.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- 172.21.15.153:0/1140630894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08481045d0 msgr2=0x7f08481049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:59.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 --2- 172.21.15.153:0/1140630894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08481045d0 0x7f08481049a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0838009880 tx=0x7f083802bce0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- 172.21.15.153:0/1140630894 shutdown_connections 2024-05-06T14:31:59.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 --2- 172.21.15.153:0/1140630894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08480fe5a0 0x7f08480fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 --2- 172.21.15.153:0/1140630894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08481045d0 0x7f08481049a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- 172.21.15.153:0/1140630894 >> 172.21.15.153:0/1140630894 conn(0x7f0848075270 msgr2=0x7f0848075670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- 172.21.15.153:0/1140630894 shutdown_connections 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- 172.21.15.153:0/1140630894 wait complete. 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 Processor -- start 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- start start 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 0x7f0848196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08481045d0 0x7f0848196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848196b40 con 0x7f08480fe5a0 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.450+0000 7f084dfda640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848196cb0 con 0x7f08481045d0 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 0x7f0848196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 0x7f0848196030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53244/0 (socket says 172.21.15.153:53244) 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 -- 172.21.15.153:0/3554661290 learned_addr learned my addr 172.21.15.153:0/3554661290 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f083ffff640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08481045d0 0x7f0848196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 -- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08481045d0 msgr2=0x7f0848196570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08481045d0 0x7f0848196570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 -- 172.21.15.153:0/3554661290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838009560 con 0x7f08480fe5a0 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f083ffff640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08481045d0 0x7f0848196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08477fe640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 0x7f0848196030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0838009230 tx=0x7f0838004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0838004420 con 0x7f08480fe5a0 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084819f0a0 con 0x7f08480fe5a0 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f08380054b0 con 0x7f08480fe5a0 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0838003e10 con 0x7f08480fe5a0 2024-05-06T14:31:59.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.451+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084819f4c0 con 0x7f08480fe5a0 2024-05-06T14:31:59.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.453+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084804e000 con 0x7f08480fe5a0 2024-05-06T14:31:59.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.453+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f0838003680 con 0x7f08480fe5a0 2024-05-06T14:31:59.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.453+0000 7f08457fa640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f083406d3d0 0x7f083406f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:59.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.453+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f08380aefc0 con 0x7f08480fe5a0 2024-05-06T14:31:59.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.454+0000 7f083ffff640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f083406d3d0 0x7f083406f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:59.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.454+0000 7f083ffff640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f083406d3d0 0x7f083406f880 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0848067bc0 tx=0x7f0830005f90 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:59.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.458+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f08380788c0 con 0x7f08480fe5a0 2024-05-06T14:31:59.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:59 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:31:59.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:59 smithi153 ceph-mon[27177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T14:31:59.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:59 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:59.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:59 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:59.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:59 smithi153 ceph-mon[27177]: Upgrade: Need to upgrade myself (mgr.smithi190.qauohf) 2024-05-06T14:31:59.575 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:31:59 smithi153 ceph-mon[27177]: Upgrade: Need to upgrade myself (mgr.smithi190.qauohf) 2024-05-06T14:31:59.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.685+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f084810ba40 con 0x7f08480fe5a0 2024-05-06T14:31:59.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.687+0000 7f08457fa640 1 -- 172.21.15.153:0/3554661290 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+15543 (secure 0 0 0) 0x7f084810ba40 con 0x7f08480fe5a0 2024-05-06T14:31:59.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-05-06T14:31:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":49,"fsid":"eacfcfb6-0bb3-11ef-bc97-c7b262605968","created":"2024-05-06T14:22:12.905455+0000","modified":"2024-05-06T14:31:19.297414+0000","last_up_change":"2024-05-06T14:27:54.584488+0000","last_in_change":"2024-05-06T14:27:35.267866+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T14:26:41.240977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-05-06T14:29:08.798039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-05-06T14:29:09.557951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"de83eb29-8176-4eb5-9be6-64b9e5449892","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6803","nonce":908509767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6805","nonce":908509767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6809","nonce":908509767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":908509767},{"type":"v1","addr":"172.21.15.153:6807","nonce":908509767}]},"public_addr":"172.21.15.153:6803/908509767","cluster_addr":"172.21.15.153:6805/908509767","heartbeat_back_addr":"172.21.15.153:6809/908509767","heartbeat_front_addr":"172.21.15.153:6807/908509767","state":["exists","up"]},{"osd":1,"uuid":"24a1c7ac-a168-4e7d-8c34-3e4c73408756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6811","nonce":3716718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6813","nonce":3716718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6817","nonce":3716718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3716718209},{"type":"v1","addr":"172.21.15.153:6815","nonce":3716718209}]},"public_addr":"172.21.15.153:6811/3716718209","cluster_addr":"172.21.15.153:6813/3716718209","heartbeat_back_addr":"172.21.15.153:6817/3716718209","heartbeat_front_addr":"172.21.15.153:6815/3716718209","state":["exists","up"]},{"osd":2,"uuid":"5b89edb2-1c0e-4481-91ed-157d767c0c05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6819","nonce":3169611803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6821","nonce":3169611803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6825","nonce":3169611803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":3169611803},{"type":"v1","addr":"172.21.15.153:6823","nonce":3169611803}]},"public_addr":"172.21.15.153:6819/3169611803","cluster_addr":"172.21.15.153:6821/3169611803","heartbeat_back_addr":"172.21.15.153:6825/3169611803","heartbeat_front_addr":"172.21.15.153:6823/3169611803","state":["exists","up"]},{"osd":3,"uuid":"74b7823e-2b78-4570-9b7a-89abc9d7fbb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6801","nonce":492034660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6803","nonce":492034660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6807","nonce":492034660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":492034660},{"type":"v1","addr":"172.21.15.190:6805","nonce":492034660}]},"public_addr":"172.21.15.190:6801/492034660","cluster_addr":"172.21.15.190:6803/492034660","heartbeat_back_addr":"172.21.15.190:6807/492034660","heartbeat_front_addr":"172.21.15.190:6805/492034660","state":["exists","up"]},{"osd":4,"uuid":"01290c44-3bf1-4da3-bac3-4dff2453cedb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6809","nonce":1342651346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6811","nonce":1342651346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6815","nonce":1342651346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1342651346},{"type":"v1","addr":"172.21.15.190:6813","nonce":1342651346}]},"public_addr":"172.21.15.190:6809/1342651346","cluster_addr":"172.21.15.190:6811/1342651346","heartbeat_back_addr":"172.21.15.190:6815/1342651346","heartbeat_front_addr":"172.21.15.190:6813/1342651346","state":["exists","up"]},{"osd":5,"uuid":"be366c2e-63d4-4496-b401-772dbe1a2444","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6817","nonce":452463735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6819","nonce":452463735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6823","nonce":452463735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":452463735},{"type":"v1","addr":"172.21.15.190:6821","nonce":452463735}]},"public_addr":"172.21.15.190:6817/452463735","cluster_addr":"172.21.15.190:6819/452463735","heartbeat_back_addr":"172.21.15.190:6823/452463735","heartbeat_front_addr":"172.21.15.190:6821/452463735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:25:42.510404+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:11.276096+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:26:37.855142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:01.189580+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:26.756150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T14:27:52.720929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6801/1473246849":"2024-05-07T14:31:18.651849+0000","172.21.15.153:6800/1473246849":"2024-05-07T14:31:18.651849+0000","172.21.15.153:0/3340397168":"2024-05-07T14:31:18.651849+0000","172.21.15.153:0/1418596404":"2024-05-07T14:31:18.651849+0000","172.21.15.153:0/3898448400":"2024-05-07T14:31:18.651849+0000","172.21.15.153:6827/4248251632":"2024-05-07T14:29:26.017160+0000","172.21.15.153:0/3158057303":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6826/4248251632":"2024-05-07T14:29:26.017160+0000","172.21.15.153:6801/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/3536681135":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1199411437":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6800/4102256583":"2024-05-07T14:22:40.569350+0000","172.21.15.153:0/1309940234":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2948592992":"2024-05-07T14:23:12.780977+0000","172.21.15.153:6800/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/4101428691":"2024-05-07T14:31:18.651849+0000","172.21.15.153:6801/585874296":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/1311303452":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/2161963253":"2024-05-07T14:22:40.569350+0000","172.21.15.153:6801/2228103355":"2024-05-07T14:24:24.923607+0000","172.21.15.153:0/1460873727":"2024-05-07T14:23:12.780977+0000","172.21.15.153:0/2433042994":"2024-05-07T14:24:24.923607+0000","172.21.15.153:6800/2228103355":"2024-05-07T14:24:24.923607+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T14:31:59.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f083406d3d0 msgr2=0x7f083406f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:59.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f083406d3d0 0x7f083406f880 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0848067bc0 tx=0x7f0830005f90 comp rx=0 tx=0).stop 2024-05-06T14:31:59.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 msgr2=0x7f0848196030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:59.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 0x7f0848196030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0838009230 tx=0x7f0838004a10 comp rx=0 tx=0).stop 2024-05-06T14:31:59.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 shutdown_connections 2024-05-06T14:31:59.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f083406d3d0 0x7f083406f880 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08481045d0 0x7f0848196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 --2- 172.21.15.153:0/3554661290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08480fe5a0 0x7f0848196030 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 >> 172.21.15.153:0/3554661290 conn(0x7f0848075270 msgr2=0x7f08480fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:59.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 shutdown_connections 2024-05-06T14:31:59.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.690+0000 7f084dfda640 1 -- 172.21.15.153:0/3554661290 wait complete. 2024-05-06T14:31:59.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:59 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T14:31:59.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:59 smithi190 ceph-mon[33671]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T14:31:59.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:59 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:31:59.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:59 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:31:59.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:59 smithi190 ceph-mon[33671]: Upgrade: Need to upgrade myself (mgr.smithi190.qauohf) 2024-05-06T14:31:59.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:31:59 smithi190 ceph-mon[33671]: Upgrade: Need to upgrade myself (mgr.smithi190.qauohf) 2024-05-06T14:31:59.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T14:26:41.240977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-05-06T14:29:08.798039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-05-06T14:29:09.557951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-06T14:31:59.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-06T14:31:59.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.813+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2720022402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 msgr2=0x7ff344104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:59.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.813+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2720022402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344104960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff340009880 tx=0x7ff34002bce0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.813+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2720022402 shutdown_connections 2024-05-06T14:31:59.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.813+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2720022402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 0x7ff3440fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.813+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2720022402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344104960 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.813+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2720022402 >> 172.21.15.153:0/2720022402 conn(0x7ff3440743d0 msgr2=0x7ff3440747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:31:59.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2720022402 shutdown_connections 2024-05-06T14:31:59.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2720022402 wait complete. 2024-05-06T14:31:59.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 Processor -- start 2024-05-06T14:31:59.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 -- start start 2024-05-06T14:31:59.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 0x7ff344195330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:59.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:59.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff344195f00 con 0x7ff344104590 2024-05-06T14:31:59.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff34bd5c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff344199cd0 con 0x7ff3440fe5b0 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.814+0000 7ff3492f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff3492f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53258/0 (socket says 172.21.15.153:53258) 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff3492f7640 1 -- 172.21.15.153:0/2998049556 learned_addr learned my addr 172.21.15.153:0/2998049556 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff349af8640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 0x7ff344195330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff3492f7640 1 -- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 msgr2=0x7ff344195330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff3492f7640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 0x7ff344195330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff3492f7640 1 -- 172.21.15.153:0/2998049556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff340009560 con 0x7ff344104590 2024-05-06T14:31:59.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff349af8640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 0x7ff344195330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff3492f7640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344195870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff33400e970 tx=0x7ff33400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff33400cc90 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344199fb0 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff33400cdf0 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff334010430 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.815+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34419a510 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.816+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34404e000 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.817+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7ff3340046c0 con 0x7ff344104590 2024-05-06T14:31:59.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.817+0000 7ff33affd640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7ff33006d330 0x7ff33006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:31:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.817+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7ff334014070 con 0x7ff344104590 2024-05-06T14:31:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.818+0000 7ff349af8640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7ff33006d330 0x7ff33006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:31:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.818+0000 7ff349af8640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7ff33006d330 0x7ff33006f7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff340006010 tx=0x7ff340005b50 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:31:59.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:31:59.826+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7ff334056a10 con 0x7ff344104590 2024-05-06T14:32:00.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.017+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff34410b860 con 0x7ff344104590 2024-05-06T14:32:00.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.018+0000 7ff33affd640 1 -- 172.21.15.153:0/2998049556 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff33405a0d0 con 0x7ff344104590 2024-05-06T14:32:00.006 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-05-06T14:32:00.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7ff33006d330 msgr2=0x7ff33006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7ff33006d330 0x7ff33006f7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff340006010 tx=0x7ff340005b50 comp rx=0 tx=0).stop 2024-05-06T14:32:00.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 msgr2=0x7ff344195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344195870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff33400e970 tx=0x7ff33400ed70 comp rx=0 tx=0).stop 2024-05-06T14:32:00.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 shutdown_connections 2024-05-06T14:32:00.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7ff33006d330 0x7ff33006f7e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff344104590 0x7ff344195870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 --2- 172.21.15.153:0/2998049556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3440fe5b0 0x7ff344195330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 >> 172.21.15.153:0/2998049556 conn(0x7ff3440743d0 msgr2=0x7ff344067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:00.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 shutdown_connections 2024-05-06T14:32:00.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.020+0000 7ff34bd5c640 1 -- 172.21.15.153:0/2998049556 wait complete. 2024-05-06T14:32:00.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-05-06T14:32:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 -- 172.21.15.153:0/741663913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690067350 msgr2=0x7f96900677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 --2- 172.21.15.153:0/741663913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690067350 0x7f96900677c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9680009880 tx=0x7f968002bce0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 -- 172.21.15.153:0/741663913 shutdown_connections 2024-05-06T14:32:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 --2- 172.21.15.153:0/741663913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690067350 0x7f96900677c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 --2- 172.21.15.153:0/741663913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690105690 0x7f9690105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 -- 172.21.15.153:0/741663913 >> 172.21.15.153:0/741663913 conn(0x7f96900743d0 msgr2=0x7f96900747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 -- 172.21.15.153:0/741663913 shutdown_connections 2024-05-06T14:32:00.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.135+0000 7f969673c640 1 -- 172.21.15.153:0/741663913 wait complete. 2024-05-06T14:32:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f969673c640 1 Processor -- start 2024-05-06T14:32:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f969673c640 1 -- start start 2024-05-06T14:32:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f969673c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690067350 0x7f9690195340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f969673c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 0x7f9690195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f969673c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9690195f10 con 0x7f9690105690 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f969673c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9690199ce0 con 0x7f9690067350 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 0x7f9690195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 0x7f9690195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53272/0 (socket says 172.21.15.153:53272) 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968f7fe640 1 -- 172.21.15.153:0/1307203923 learned_addr learned my addr 172.21.15.153:0/1307203923 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968ffff640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690067350 0x7f9690195340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968f7fe640 1 -- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690067350 msgr2=0x7f9690195340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968f7fe640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690067350 0x7f9690195340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968f7fe640 1 -- 172.21.15.153:0/1307203923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9680009560 con 0x7f9690105690 2024-05-06T14:32:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.136+0000 7f968ffff640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690067350 0x7f9690195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-06T14:32:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.137+0000 7f968f7fe640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 0x7f9690195880 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9680009230 tx=0x7f9680004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.137+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9680004420 con 0x7f9690105690 2024-05-06T14:32:00.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.137+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9690199f60 con 0x7f9690105690 2024-05-06T14:32:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.137+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f96800054b0 con 0x7f9690105690 2024-05-06T14:32:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.137+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9680003e10 con 0x7f9690105690 2024-05-06T14:32:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.137+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969019a460 con 0x7f9690105690 2024-05-06T14:32:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.138+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969004e000 con 0x7f9690105690 2024-05-06T14:32:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.138+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f9680003680 con 0x7f9690105690 2024-05-06T14:32:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.139+0000 7f968d7fa640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f967806d330 0x7f967806f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.139+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f96800af010 con 0x7f9690105690 2024-05-06T14:32:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.139+0000 7f968ffff640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f967806d330 0x7f967806f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.140+0000 7f968ffff640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f967806d330 0x7f967806f7e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9684009c20 tx=0x7f9684005f90 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:00.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.145+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f9680078860 con 0x7f9690105690 2024-05-06T14:32:00.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.373+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f9690107aa0 con 0x7f9690105690 2024-05-06T14:32:00.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.373+0000 7f968d7fa640 1 -- 172.21.15.153:0/1307203923 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v49) v1 ==== 107+0+11 (secure 0 0 0) 0x7f968009c070 con 0x7f9690105690 2024-05-06T14:32:00.361 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-05-06T14:32:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f967806d330 msgr2=0x7f967806f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f967806d330 0x7f967806f7e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9684009c20 tx=0x7f9684005f90 comp rx=0 tx=0).stop 2024-05-06T14:32:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 msgr2=0x7f9690195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 0x7f9690195880 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9680009230 tx=0x7f9680004a10 comp rx=0 tx=0).stop 2024-05-06T14:32:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 shutdown_connections 2024-05-06T14:32:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f967806d330 0x7f967806f7e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9690105690 0x7f9690195880 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 --2- 172.21.15.153:0/1307203923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9690067350 0x7f9690195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 >> 172.21.15.153:0/1307203923 conn(0x7f96900743d0 msgr2=0x7f96900fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:00.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 shutdown_connections 2024-05-06T14:32:00.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.375+0000 7f969673c640 1 -- 172.21.15.153:0/1307203923 wait complete. 2024-05-06T14:32:00.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-05-06T14:32:00.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.494+0000 7f437c650640 1 -- 172.21.15.153:0/99374206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 msgr2=0x7f43740ff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.494+0000 7f437c650640 1 --2- 172.21.15.153:0/99374206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43740ff140 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4364009880 tx=0x7f436402bce0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 -- 172.21.15.153:0/99374206 shutdown_connections 2024-05-06T14:32:00.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 --2- 172.21.15.153:0/99374206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43740ff140 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 --2- 172.21.15.153:0/99374206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4374104590 0x7f4374104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 -- 172.21.15.153:0/99374206 >> 172.21.15.153:0/99374206 conn(0x7f43740749d0 msgr2=0x7f4374074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:00.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 -- 172.21.15.153:0/99374206 shutdown_connections 2024-05-06T14:32:00.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 -- 172.21.15.153:0/99374206 wait complete. 2024-05-06T14:32:00.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 Processor -- start 2024-05-06T14:32:00.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.495+0000 7f437c650640 1 -- start start 2024-05-06T14:32:00.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437c650640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43741960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437c650640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4374104590 0x7f4374196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437c650640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374196d10 con 0x7f43740fecd0 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437c650640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374196e50 con 0x7f4374104590 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43741960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43741960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53288/0 (socket says 172.21.15.153:53288) 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 -- 172.21.15.153:0/401365308 learned_addr learned my addr 172.21.15.153:0/401365308 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f4379beb640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4374104590 0x7f4374196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 -- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4374104590 msgr2=0x7f4374196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4374104590 0x7f4374196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 -- 172.21.15.153:0/401365308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364009560 con 0x7f43740fecd0 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.496+0000 7f437a3ec640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43741960f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f437000d6f0 tx=0x7f437000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.497+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4370009a30 con 0x7f43740fecd0 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.497+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f43700054d0 con 0x7f43740fecd0 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.497+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437419f160 con 0x7f43740fecd0 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.497+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4370010460 con 0x7f43740fecd0 2024-05-06T14:32:00.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.497+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437419f690 con 0x7f43740fecd0 2024-05-06T14:32:00.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.498+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437404e000 con 0x7f43740fecd0 2024-05-06T14:32:00.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.498+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f4370009b90 con 0x7f43740fecd0 2024-05-06T14:32:00.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.499+0000 7f436b7fe640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f436006d330 0x7f436006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.499+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f437008c700 con 0x7f43740fecd0 2024-05-06T14:32:00.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.499+0000 7f4379beb640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f436006d330 0x7f436006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.500+0000 7f4379beb640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f436006d330 0x7f436006f7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4364006010 tx=0x7f4364005dd0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:00.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.507+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f4370055f70 con 0x7f43740fecd0 2024-05-06T14:32:00.628 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:00 smithi153 ceph-mon[27177]: pgmap v23: 65 pgs: 65 active+clean; 193 MiB data, 901 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s wr, 183 op/s 2024-05-06T14:32:00.629 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:00 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/3554661290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:32:00.629 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:00 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2998049556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:32:00.629 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:00 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/1307203923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-06T14:32:00.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:00 smithi190 ceph-mon[33671]: pgmap v23: 65 pgs: 65 active+clean; 193 MiB data, 901 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s wr, 183 op/s 2024-05-06T14:32:00.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:00 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/3554661290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T14:32:00.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:00 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2998049556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T14:32:00.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:00 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/1307203923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-05-06T14:32:00.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.770+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f43741069d0 con 0x7f43740fecd0 2024-05-06T14:32:00.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.770+0000 7f436b7fe640 1 -- 172.21.15.153:0/401365308 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v49) v1 ==== 107+0+11 (secure 0 0 0) 0x7f43741069d0 con 0x7f43740fecd0 2024-05-06T14:32:00.759 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-05-06T14:32:00.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f436006d330 msgr2=0x7f436006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f436006d330 0x7f436006f7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4364006010 tx=0x7f4364005dd0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 msgr2=0x7f43741960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43741960f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f437000d6f0 tx=0x7f437000dbb0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 shutdown_connections 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f436006d330 0x7f436006f7e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4374104590 0x7f4374196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.772+0000 7f437c650640 1 --2- 172.21.15.153:0/401365308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43740fecd0 0x7f43741960f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.773+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 >> 172.21.15.153:0/401365308 conn(0x7f43740749d0 msgr2=0x7f43740fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:00.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.773+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 shutdown_connections 2024-05-06T14:32:00.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.773+0000 7f437c650640 1 -- 172.21.15.153:0/401365308 wait complete. 2024-05-06T14:32:00.774 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-06T14:32:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.891+0000 7f40149c3640 1 -- 172.21.15.153:0/3453785465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 msgr2=0x7f40101095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.891+0000 7f40149c3640 1 --2- 172.21.15.153:0/3453785465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f40101095f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80098b0 tx=0x7f3ff802bd60 comp rx=0 tx=0).stop 2024-05-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.891+0000 7f40149c3640 1 -- 172.21.15.153:0/3453785465 shutdown_connections 2024-05-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.891+0000 7f40149c3640 1 --2- 172.21.15.153:0/3453785465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f40101095f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.891+0000 7f40149c3640 1 --2- 172.21.15.153:0/3453785465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40101007f0 0x7f4010100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 -- 172.21.15.153:0/3453785465 >> 172.21.15.153:0/3453785465 conn(0x7f4010075250 msgr2=0x7f4010075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 -- 172.21.15.153:0/3453785465 shutdown_connections 2024-05-06T14:32:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 -- 172.21.15.153:0/3453785465 wait complete. 2024-05-06T14:32:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 Processor -- start 2024-05-06T14:32:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 -- start start 2024-05-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40101007f0 0x7f4010195230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f4010195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.892+0000 7f40149c3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4010195e00 con 0x7f4010101100 2024-05-06T14:32:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f40149c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4010199bd0 con 0x7f40101007f0 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f4010195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f4010195770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53310/0 (socket says 172.21.15.153:53310) 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 -- 172.21.15.153:0/2209771591 learned_addr learned my addr 172.21.15.153:0/2209771591 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f400e59c640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40101007f0 0x7f4010195230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 -- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40101007f0 msgr2=0x7f4010195230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40101007f0 0x7f4010195230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 -- 172.21.15.153:0/2209771591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8009560 con 0x7f4010101100 2024-05-06T14:32:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f4007fff640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f4010195770 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009880 tx=0x7f3ff802c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ff80044e0 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ff80054b0 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4010199e50 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ff803e450 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.893+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401019a3b0 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.895+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401004e000 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.895+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f3ff8004f20 con 0x7f4010101100 2024-05-06T14:32:00.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.895+0000 7f40077fe640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f3ffc06d3d0 0x7f3ffc06f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.896+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f3ff80af820 con 0x7f4010101100 2024-05-06T14:32:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.896+0000 7f400e59c640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f3ffc06d3d0 0x7f3ffc06f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.896+0000 7f400e59c640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f3ffc06d3d0 0x7f3ffc06f880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4000005fd0 tx=0x7f4000005850 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:00.900+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f3ff8079070 con 0x7f4010101100 2024-05-06T14:32:01.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.100+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4010106c60 con 0x7f4010101100 2024-05-06T14:32:01.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.100+0000 7f40077fe640 1 -- 172.21.15.153:0/2209771591 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 23 entries v49) v1 ==== 81+0+1385 (secure 0 0 0) 0x7f3ff8043580 con 0x7f4010101100 2024-05-06T14:32:01.088 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/1473246849 2024-05-07T14:31:18.651849+0000 2024-05-06T14:32:01.089 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/1473246849 2024-05-07T14:31:18.651849+0000 2024-05-06T14:32:01.089 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3340397168 2024-05-07T14:31:18.651849+0000 2024-05-06T14:32:01.089 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1418596404 2024-05-07T14:31:18.651849+0000 2024-05-06T14:32:01.089 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3898448400 2024-05-07T14:31:18.651849+0000 2024-05-06T14:32:01.089 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6827/4248251632 2024-05-07T14:29:26.017160+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3158057303 2024-05-07T14:24:24.923607+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6826/4248251632 2024-05-07T14:29:26.017160+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/4102256583 2024-05-07T14:22:40.569350+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3536681135 2024-05-07T14:22:40.569350+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1199411437 2024-05-07T14:22:40.569350+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/4102256583 2024-05-07T14:22:40.569350+0000 2024-05-06T14:32:01.090 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1309940234 2024-05-07T14:23:12.780977+0000 2024-05-06T14:32:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2948592992 2024-05-07T14:23:12.780977+0000 2024-05-06T14:32:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/585874296 2024-05-07T14:23:12.780977+0000 2024-05-06T14:32:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/4101428691 2024-05-07T14:31:18.651849+0000 2024-05-06T14:32:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/585874296 2024-05-07T14:23:12.780977+0000 2024-05-06T14:32:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1311303452 2024-05-07T14:24:24.923607+0000 2024-05-06T14:32:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2161963253 2024-05-07T14:22:40.569350+0000 2024-05-06T14:32:01.092 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/2228103355 2024-05-07T14:24:24.923607+0000 2024-05-06T14:32:01.092 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1460873727 2024-05-07T14:23:12.780977+0000 2024-05-06T14:32:01.092 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2433042994 2024-05-07T14:24:24.923607+0000 2024-05-06T14:32:01.092 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/2228103355 2024-05-07T14:24:24.923607+0000 2024-05-06T14:32:01.093 INFO:teuthology.orchestra.run.smithi153.stderr:listed 23 entries 2024-05-06T14:32:01.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f3ffc06d3d0 msgr2=0x7f3ffc06f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:01.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f3ffc06d3d0 0x7f3ffc06f880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4000005fd0 tx=0x7f4000005850 comp rx=0 tx=0).stop 2024-05-06T14:32:01.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 msgr2=0x7f4010195770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:01.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f4010195770 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009880 tx=0x7f3ff802c770 comp rx=0 tx=0).stop 2024-05-06T14:32:01.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 shutdown_connections 2024-05-06T14:32:01.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f3ffc06d3d0 0x7f3ffc06f880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:01.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4010101100 0x7f4010195770 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:01.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 --2- 172.21.15.153:0/2209771591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40101007f0 0x7f4010195230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:01.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.102+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 >> 172.21.15.153:0/2209771591 conn(0x7f4010075250 msgr2=0x7f4010103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:01.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.103+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 shutdown_connections 2024-05-06T14:32:01.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:01.103+0000 7f40149c3640 1 -- 172.21.15.153:0/2209771591 wait complete. 2024-05-06T14:32:01.101 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-06T14:32:01.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:01.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-05-06T14:32:01.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:01 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/401365308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-06T14:32:01.435 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:01 smithi190 ceph-mon[33671]: from='client.? 172.21.15.153:0/2209771591' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:32:01.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:01 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/401365308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-05-06T14:32:01.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:01 smithi153 ceph-mon[27177]: from='client.? 172.21.15.153:0/2209771591' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:32:02.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:02 smithi190 ceph-mon[33671]: pgmap v24: 65 pgs: 65 active+clean; 242 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 7.6 MiB/s wr, 623 op/s 2024-05-06T14:32:02.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:02 smithi153 ceph-mon[27177]: pgmap v24: 65 pgs: 65 active+clean; 242 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 7.6 MiB/s wr, 623 op/s 2024-05-06T14:32:04.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: pgmap v25: 65 pgs: 65 active+clean; 242 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 6.3 MiB/s wr, 602 op/s 2024-05-06T14:32:04.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: Upgrade: Updating mgr.smithi153.dolfed 2024-05-06T14:32:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:32:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.dolfed", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:32:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:32:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:32:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:04 smithi153 ceph-mon[27177]: Deploying daemon mgr.smithi153.dolfed on smithi153 2024-05-06T14:32:04.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: pgmap v25: 65 pgs: 65 active+clean; 242 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 6.3 MiB/s wr, 602 op/s 2024-05-06T14:32:04.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: Upgrade: Updating mgr.smithi153.dolfed 2024-05-06T14:32:04.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' 2024-05-06T14:32:04.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.dolfed", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T14:32:04.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T14:32:04.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:32:04.936 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:04 smithi190 ceph-mon[33671]: Deploying daemon mgr.smithi153.dolfed on smithi153 2024-05-06T14:32:05.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:05 smithi153 ceph-mon[27177]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:32:05.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:05 smithi190 ceph-mon[33671]: from='mgr.24373 172.21.15.190:0/1882862101' entity='mgr.smithi190.qauohf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:32:06.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:06 smithi153 ceph-mon[27177]: pgmap v26: 65 pgs: 65 active+clean; 242 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 6.3 MiB/s wr, 602 op/s 2024-05-06T14:32:06.643 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi190.front.sepia.ceph.com 2024-05-06T14:32:06.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:06.643 DEBUG:teuthology.orchestra.run.smithi190:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-06T14:32:06.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:06.674 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-05-06T14:32:06.691 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-06T14:32:06.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:06.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-06T14:32:06.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:06.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-05-06T14:32:06.935 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:06 smithi190 ceph-mon[33671]: pgmap v26: 65 pgs: 65 active+clean; 242 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 6.3 MiB/s wr, 602 op/s 2024-05-06T14:32:07.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:07.085 DEBUG:teuthology.orchestra.run.smithi190:> ip netns list 2024-05-06T14:32:07.100 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-06T14:32:07.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:07.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-06T14:32:07.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-06T14:32:07.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo ip link delete ceph-brx 2024-05-06T14:32:07.251 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-06T14:32:07.266 INFO:tasks.cephadm:Teardown begin 2024-05-06T14:32:07.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:32:07.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:32:07.318 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-06T14:32:07.319 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-06T14:32:07.349 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-06T14:32:07.373 INFO:tasks.cephadm:Stopping all daemons... 2024-05-06T14:32:07.374 INFO:tasks.cephadm.mon.smithi153:Stopping mon.smithi153... 2024-05-06T14:32:07.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153 2024-05-06T14:32:07.590 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:07 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:07.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 -- 172.21.15.153:0/3473112944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 msgr2=0x7f2e1c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:07.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3473112944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e1c1058e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009870 tx=0x7f2e10009b80 comp rx=0 tx=0).stop 2024-05-06T14:32:07.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 -- 172.21.15.153:0/3473112944 shutdown_connections 2024-05-06T14:32:07.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3473112944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e1c1058e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:07.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3473112944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:07.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 -- 172.21.15.153:0/3473112944 >> 172.21.15.153:0/3473112944 conn(0x7f2e1c0fa600 msgr2=0x7f2e1c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:07.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 -- 172.21.15.153:0/3473112944 shutdown_connections 2024-05-06T14:32:07.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.796+0000 7f2e2204c700 1 -- 172.21.15.153:0/3473112944 wait complete. 2024-05-06T14:32:07.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e2204c700 1 Processor -- start 2024-05-06T14:32:07.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e2204c700 1 -- start start 2024-05-06T14:32:07.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e2204c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:07.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e2204c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e1c0feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:07.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e2204c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c1995c0 con 0x7f2e1c100be0 2024-05-06T14:32:07.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e2204c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c199700 con 0x7f2e1c103500 2024-05-06T14:32:07.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e1b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:07.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e1b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53340/0 (socket says 172.21.15.153:53340) 2024-05-06T14:32:07.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.797+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 learned_addr learned my addr 172.21.15.153:0/3301172983 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:07.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.798+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 msgr2=0x7f2e1c0feb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:07.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.798+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e1c0feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:07.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.798+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10009520 con 0x7f2e1c100be0 2024-05-06T14:32:07.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.798+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00e970 tx=0x7f2e0c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:07.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.799+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c00c9d0 con 0x7f2e1c100be0 2024-05-06T14:32:07.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.799+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c0ff0d0 con 0x7f2e1c100be0 2024-05-06T14:32:07.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.799+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c0ff670 con 0x7f2e1c100be0 2024-05-06T14:32:07.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.799+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2e0c004830 con 0x7f2e1c100be0 2024-05-06T14:32:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.799+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c005690 con 0x7f2e1c100be0 2024-05-06T14:32:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.800+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c18fb90 con 0x7f2e1c100be0 2024-05-06T14:32:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.801+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f2e0c00cb30 con 0x7f2e1c100be0 2024-05-06T14:32:07.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.801+0000 7f2e18ff9700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2e0406d3c0 0x7f2e0406f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.802+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f2e0c014070 con 0x7f2e1c100be0 2024-05-06T14:32:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.802+0000 7f2e1affd700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2e0406d3c0 0x7f2e0406f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:07.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.802+0000 7f2e1affd700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2e0406d3c0 0x7f2e0406f870 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009230 tx=0x7f2e10009ef0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:07.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:07.806+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f2e0c05a3c0 con 0x7f2e1c100be0 2024-05-06T14:32:08.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 msgr2=0x7f2e1c100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-06T14:32:08.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 msgr2=0x7f2e1c100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-06T14:32:08.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00e970 tx=0x7f2e0c00ed30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00e970 tx=0x7f2e0c00ed30 comp rx=0 tx=0).stop 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 msgr2=0x7f2e1c100540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e18ff9700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1c100be0 0x7f2e1c100540 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e18ff9700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e04076da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e18ff9700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04077550 0x7f2e040799d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0407a3f0 con 0x7f2e04077550 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0407a530 con 0x7f2e1c103500 2024-05-06T14:32:08.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e1bfff700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04077550 msgr2=0x7f2e040799d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-05-06T14:32:08.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.058+0000 7f2e1bfff700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04077550 0x7f2e040799d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:32:08.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.059+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e04076da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.060+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04077550 msgr2=0x7f2e040799d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:32:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.060+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04077550 0x7f2e040799d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.060+0000 7f2e1b7fe700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7f2e0407a730 con 0x7f2e1c103500 2024-05-06T14:32:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.060+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e1c100540 con 0x7f2e0406d3c0 2024-05-06T14:32:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.061+0000 7f2e1b7fe700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e04076da0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c08b230 tx=0x7f2e0c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.061+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0c05e9a0 con 0x7f2e1c103500 2024-05-06T14:32:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.061+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2e0c0631d0 con 0x7f2e1c103500 2024-05-06T14:32:08.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.063+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f2e0c08b420 con 0x7f2e1c103500 2024-05-06T14:32:08.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.064+0000 7f2e18ff9700 1 -- 172.21.15.153:0/3301172983 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f2e1c100540 con 0x7f2e0406d3c0 2024-05-06T14:32:08.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2e0406d3c0 msgr2=0x7f2e0406f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2e0406d3c0 0x7f2e0406f870 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009230 tx=0x7f2e10009ef0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 msgr2=0x7f2e04076da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e04076da0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c08b230 tx=0x7f2e0c009310 comp rx=0 tx=0).stop 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 shutdown_connections 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e04077550 0x7f2e040799d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e1c103500 0x7f2e04076da0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 --2- 172.21.15.153:0/3301172983 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f2e0406d3c0 0x7f2e0406f870 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 >> 172.21.15.153:0/3301172983 conn(0x7f2e1c0fa600 msgr2=0x7f2e1c0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:08.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 shutdown_connections 2024-05-06T14:32:08.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.067+0000 7f2e2204c700 1 -- 172.21.15.153:0/3301172983 wait complete. 2024-05-06T14:32:08.071 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 -- 172.21.15.153:0/2209303374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 msgr2=0x7fec74100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/2209303374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74100a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec70009910 tx=0x7fec70009c20 comp rx=0 tx=0).stop 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 -- 172.21.15.153:0/2209303374 shutdown_connections 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/2209303374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 0x7fec740ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/2209303374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74100a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 -- 172.21.15.153:0/2209303374 >> 172.21.15.153:0/2209303374 conn(0x7fec740f9d80 msgr2=0x7fec740fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:08.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 -- 172.21.15.153:0/2209303374 shutdown_connections 2024-05-06T14:32:08.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.210+0000 7fec7bf8e700 1 -- 172.21.15.153:0/2209303374 wait complete. 2024-05-06T14:32:08.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec7bf8e700 1 Processor -- start 2024-05-06T14:32:08.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec7bf8e700 1 -- start start 2024-05-06T14:32:08.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec7bf8e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 0x7fec740fecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec7bf8e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec7bf8e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74198ae0 con 0x7fec740ff260 2024-05-06T14:32:08.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec7bf8e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74198c50 con 0x7fec74100600 2024-05-06T14:32:08.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec79d2a700 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 msgr2=0x7fec740fecd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-05-06T14:32:08.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec79d2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 0x7fec740fecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.211+0000 7fec79529700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec79529700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:56116/0 (socket says 172.21.15.153:56116) 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec79529700 1 -- 172.21.15.153:0/3159008641 learned_addr learned my addr 172.21.15.153:0/3159008641 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec79529700 1 -- 172.21.15.153:0/3159008641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 msgr2=0x7fec740fecd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec79529700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 0x7fec740fecd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec79529700 1 -- 172.21.15.153:0/3159008641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700095f0 con 0x7fec74100600 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec79529700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74198510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec6400b700 tx=0x7fec6400bac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec64011800 con 0x7fec74100600 2024-05-06T14:32:08.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74198f30 con 0x7fec74100600 2024-05-06T14:32:08.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fec64011e10 con 0x7fec74100600 2024-05-06T14:32:08.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec6400f2e0 con 0x7fec74100600 2024-05-06T14:32:08.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.212+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74199500 con 0x7fec74100600 2024-05-06T14:32:08.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.214+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec58005320 con 0x7fec74100600 2024-05-06T14:32:08.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.214+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7fec6400f440 con 0x7fec74100600 2024-05-06T14:32:08.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.215+0000 7fec6affd700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fec6006d300 0x7fec6006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.215+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fec6408c7b0 con 0x7fec74100600 2024-05-06T14:32:08.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.215+0000 7fec79d2a700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fec6006d300 0x7fec6006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.216+0000 7fec79d2a700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fec6006d300 0x7fec6006f7b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec70005090 tx=0x7fec70009fd0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.220+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7fec64059bb0 con 0x7fec74100600 2024-05-06T14:32:08.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:08 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153[27173]: 2024-05-06T14:32:08.057+0000 7f9b5be6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:08.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:08 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153[27173]: 2024-05-06T14:32:08.057+0000 7f9b5be6b700 -1 mon.smithi153@0(leader) e2 *** Got Signal Terminated *** 2024-05-06T14:32:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:08 smithi153 podman[70489]: 2024-05-06 14:32:08.140362151 +0000 UTC m=+0.408659404 container died 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212) 2024-05-06T14:32:08.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.511+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec58000bf0 con 0x7fec6006d300 2024-05-06T14:32:08.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.513+0000 7fec6affd700 1 -- 172.21.15.153:0/3159008641 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fec58000bf0 con 0x7fec6006d300 2024-05-06T14:32:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fec6006d300 msgr2=0x7fec6006f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fec6006d300 0x7fec6006f7b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec70005090 tx=0x7fec70009fd0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 msgr2=0x7fec74198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74198510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec6400b700 tx=0x7fec6400bac0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 shutdown_connections 2024-05-06T14:32:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fec6006d300 0x7fec6006f7b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec740ff260 0x7fec740fecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 --2- 172.21.15.153:0/3159008641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec74100600 0x7fec74198510 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 >> 172.21.15.153:0/3159008641 conn(0x7fec740f9d80 msgr2=0x7fec741034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 shutdown_connections 2024-05-06T14:32:08.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.515+0000 7fec7bf8e700 1 -- 172.21.15.153:0/3159008641 wait complete. 2024-05-06T14:32:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.625+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/3261281763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980fea30 msgr2=0x7f8a980fee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.625+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/3261281763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980fea30 0x7f8a980fee40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009910 tx=0x7f8a94009c20 comp rx=0 tx=0).stop 2024-05-06T14:32:08.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.625+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/3261281763 shutdown_connections 2024-05-06T14:32:08.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.625+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/3261281763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980ff380 0x7f8a980ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/3261281763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980fea30 0x7f8a980fee40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/3261281763 >> 172.21.15.153:0/3261281763 conn(0x7f8a980fa5e0 msgr2=0x7f8a980fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:08.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/3261281763 shutdown_connections 2024-05-06T14:32:08.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/3261281763 wait complete. 2024-05-06T14:32:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 Processor -- start 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 -- start start 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980fea30 0x7f8a98197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 0x7f8a98198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98198730 con 0x7f8a980fea30 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.626+0000 7f8a9f7b5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98198870 con 0x7f8a980ff380 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9d551700 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980fea30 msgr2=0x7f8a98197bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9d551700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980fea30 0x7f8a98197bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:32:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 0x7f8a98198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 0x7f8a98198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:56138/0 (socket says 172.21.15.153:56138) 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 -- 172.21.15.153:0/2219606949 learned_addr learned my addr 172.21.15.153:0/2219606949 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 -- 172.21.15.153:0/2219606949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980fea30 msgr2=0x7f8a98197bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980fea30 0x7f8a98197bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 -- 172.21.15.153:0/2219606949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a940095f0 con 0x7f8a980ff380 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9cd50700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 0x7f8a98198110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800b4a0 tx=0x7f8a8800b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a88011800 con 0x7f8a980ff380 2024-05-06T14:32:08.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8a88011e10 con 0x7f8a980ff380 2024-05-06T14:32:08.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9819d320 con 0x7f8a980ff380 2024-05-06T14:32:08.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.627+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a8800f2e0 con 0x7f8a980ff380 2024-05-06T14:32:08.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.628+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9819d8f0 con 0x7f8a980ff380 2024-05-06T14:32:08.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.629+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9810af20 con 0x7f8a980ff380 2024-05-06T14:32:08.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.629+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f8a88010380 con 0x7f8a980ff380 2024-05-06T14:32:08.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.629+0000 7f8a8e7fc700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f8a8406d360 0x7f8a8406f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.630+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8a8808c750 con 0x7f8a980ff380 2024-05-06T14:32:08.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.630+0000 7f8a9d551700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f8a8406d360 0x7f8a8406f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.630+0000 7f8a9d551700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f8a8406d360 0x7f8a8406f810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8a94005090 tx=0x7f8a94009fd0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.634+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f8a88059ba0 con 0x7f8a980ff380 2024-05-06T14:32:08.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.868+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a98108780 con 0x7f8a8406d360 2024-05-06T14:32:08.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.878+0000 7f8a8e7fc700 1 -- 172.21.15.153:0/2219606949 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8a98108780 con 0x7f8a8406d360 2024-05-06T14:32:08.866 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:32:08.866 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (6m) 37s ago 8m 24.9M - 0.25.0 c8568f914cd2 5869e5f8b16f 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi153 smithi153 running (8m) 37s ago 8m 7566k - 18.2.0 dc2bc1663786 9903f06d2ec9 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-exporter.smithi190 smithi190 running (7m) 26s ago 7m 7532k - 18.2.0 dc2bc1663786 8fa6eb9bb609 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (8m) 37s ago 8m 6660k - 18.2.0 dc2bc1663786 d9ddce025318 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi190 smithi190 running (7m) 26s ago 7m 6669k - 18.2.0 dc2bc1663786 1b44eaee674c 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (6m) 37s ago 8m 87.8M - 9.4.7 954c08fa6188 d15dfe6401b3 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.ftbnnk smithi153 running (2m) 37s ago 2m 81.3M - 18.2.0 dc2bc1663786 ecb0b5505691 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.hiwroq smithi153 running (2m) 37s ago 2m 12.8M - 18.2.0 dc2bc1663786 0c0228faf2c1 2024-05-06T14:32:08.867 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.psnzbz smithi190 running (2m) 26s ago 2m 18.9M - 18.2.0 dc2bc1663786 ad189e8605bd 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi190.quxdec smithi190 running (2m) 26s ago 2m 16.6M - 18.2.0 dc2bc1663786 869fb60edc82 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.dolfed smithi153 *:8443,9283,8765 starting - - - - 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi190.qauohf smithi190 *:8443,9283,8765 running (59s) 26s ago 7m 490M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 00df65339e6f 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (9m) 37s ago 9m 49.2M 2048M 18.2.0 dc2bc1663786 44f2993c90f4 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi190 smithi190 running (7m) 26s ago 7m 37.6M 2048M 18.2.0 dc2bc1663786 88aeab4dfa5e 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 37s ago 8m 15.2M - 1.5.0 0da6a335fe13 5c569e0ae0c9 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi190 smithi190 *:9100 running (7m) 26s ago 7m 16.8M - 1.5.0 0da6a335fe13 d1be4aed7b03 2024-05-06T14:32:08.868 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (6m) 37s ago 6m 72.4M 1216M 18.2.0 dc2bc1663786 6c202dfae018 2024-05-06T14:32:08.869 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (6m) 37s ago 6m 78.2M 1216M 18.2.0 dc2bc1663786 78ff3749996c 2024-05-06T14:32:08.869 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (5m) 37s ago 5m 68.0M 1216M 18.2.0 dc2bc1663786 3fbd486b6ac6 2024-05-06T14:32:08.869 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi190 running (5m) 26s ago 5m 75.7M 2240M 18.2.0 dc2bc1663786 79af272339eb 2024-05-06T14:32:08.869 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi190 running (4m) 26s ago 4m 70.6M 2240M 18.2.0 dc2bc1663786 6db95b0197f1 2024-05-06T14:32:08.869 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi190 running (4m) 26s ago 4m 71.1M 2240M 18.2.0 dc2bc1663786 ae3651b2db8d 2024-05-06T14:32:08.869 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 starting - - - - 2024-05-06T14:32:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f8a8406d360 msgr2=0x7f8a8406f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f8a8406d360 0x7f8a8406f810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8a94005090 tx=0x7f8a94009fd0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 msgr2=0x7f8a98198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 0x7f8a98198110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800b4a0 tx=0x7f8a8800b860 comp rx=0 tx=0).stop 2024-05-06T14:32:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 shutdown_connections 2024-05-06T14:32:08.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f8a8406d360 0x7f8a8406f810 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980fea30 0x7f8a98197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 --2- 172.21.15.153:0/2219606949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a980ff380 0x7f8a98198110 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 >> 172.21.15.153:0/2219606949 conn(0x7f8a980fa5e0 msgr2=0x7f8a98107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 shutdown_connections 2024-05-06T14:32:08.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.881+0000 7f8a9f7b5700 1 -- 172.21.15.153:0/2219606949 wait complete. 2024-05-06T14:32:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4090474661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c14100250 msgr2=0x7f1c141006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4090474661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c14100250 0x7f1c141006a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009910 tx=0x7f1c00009c20 comp rx=0 tx=0).stop 2024-05-06T14:32:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4090474661 shutdown_connections 2024-05-06T14:32:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4090474661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c14100250 0x7f1c141006a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4090474661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c140ff050 0x7f1c140ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4090474661 >> 172.21.15.153:0/4090474661 conn(0x7f1c140fa600 msgr2=0x7f1c140fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4090474661 shutdown_connections 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.988+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4090474661 wait complete. 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c1aff8700 1 Processor -- start 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c1aff8700 1 -- start start 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c1aff8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 0x7f1c14102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c1aff8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c14100250 0x7f1c14101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c1aff8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14101b70 con 0x7f1c14100250 2024-05-06T14:32:08.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c1aff8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14101cb0 con 0x7f1c140ff050 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c13fff700 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c14100250 msgr2=0x7f1c14101630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c13fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c14100250 0x7f1c14101630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.989+0000 7f1c18d94700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 0x7f1c14102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c18d94700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 0x7f1c14102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:56148/0 (socket says 172.21.15.153:56148) 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c18d94700 1 -- 172.21.15.153:0/4055970297 learned_addr learned my addr 172.21.15.153:0/4055970297 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c18d94700 1 -- 172.21.15.153:0/4055970297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c14100250 msgr2=0x7f1c14101630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c18d94700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c14100250 0x7f1c14101630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c18d94700 1 -- 172.21.15.153:0/4055970297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c000095f0 con 0x7f1c140ff050 2024-05-06T14:32:08.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c18d94700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 0x7f1c14102fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800b4d0 tx=0x7f1c0800b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c08011800 con 0x7f1c140ff050 2024-05-06T14:32:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c08011e10 con 0x7f1c140ff050 2024-05-06T14:32:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c14101f90 con 0x7f1c140ff050 2024-05-06T14:32:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c08010e90 con 0x7f1c140ff050 2024-05-06T14:32:08.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.990+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1419da60 con 0x7f1c140ff050 2024-05-06T14:32:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.992+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7f1c08010380 con 0x7f1c140ff050 2024-05-06T14:32:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.992+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c14107c40 con 0x7f1c140ff050 2024-05-06T14:32:08.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.992+0000 7f1c11ffb700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f1c0406d300 0x7f1c0406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.992+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f1c0808c850 con 0x7f1c140ff050 2024-05-06T14:32:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.992+0000 7f1c13fff700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f1c0406d300 0x7f1c0406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:08.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.993+0000 7f1c13fff700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f1c0406d300 0x7f1c0406f7b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1c00005090 tx=0x7f1c00009fd0 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:08.997+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7f1c08059bb0 con 0x7f1c140ff050 2024-05-06T14:32:09.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.325+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1c1419e010 con 0x7f1c140ff050 2024-05-06T14:32:09.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.326+0000 7f1c11ffb700 1 -- 172.21.15.153:0/4055970297 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7f1c08059740 con 0x7f1c140ff050 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 2 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 1, 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T14:32:09.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 6 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 4 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-05-06T14:32:09.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)": 13, 2024-05-06T14:32:09.316 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T14:32:09.316 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-05-06T14:32:09.316 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:32:09.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f1c0406d300 msgr2=0x7f1c0406f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f1c0406d300 0x7f1c0406f7b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1c00005090 tx=0x7f1c00009fd0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 msgr2=0x7f1c14102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 0x7f1c14102fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800b4d0 tx=0x7f1c0800b890 comp rx=0 tx=0).stop 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 shutdown_connections 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7f1c0406d300 0x7f1c0406f7b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c140ff050 0x7f1c14102fe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 --2- 172.21.15.153:0/4055970297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c14100250 0x7f1c14101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 >> 172.21.15.153:0/4055970297 conn(0x7f1c140fa600 msgr2=0x7f1c14068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:09.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 shutdown_connections 2024-05-06T14:32:09.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.328+0000 7f1c1aff8700 1 -- 172.21.15.153:0/4055970297 wait complete. 2024-05-06T14:32:09.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 -- 172.21.15.153:0/874992325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 msgr2=0x7fdffc1017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 --2- 172.21.15.153:0/874992325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc1017a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009910 tx=0x7fdfec009c20 comp rx=0 tx=0).stop 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 -- 172.21.15.153:0/874992325 shutdown_connections 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 --2- 172.21.15.153:0/874992325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc1017a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 --2- 172.21.15.153:0/874992325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 0x7fdffc100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 -- 172.21.15.153:0/874992325 >> 172.21.15.153:0/874992325 conn(0x7fdffc0fb6e0 msgr2=0x7fdffc0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 -- 172.21.15.153:0/874992325 shutdown_connections 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 -- 172.21.15.153:0/874992325 wait complete. 2024-05-06T14:32:09.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 Processor -- start 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.437+0000 7fe001bb2700 1 -- start start 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fe001bb2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 0x7fdffc195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fe001bb2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fe001bb2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc196570 con 0x7fdffc100150 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fe001bb2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc1966b0 con 0x7fdffc101350 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffb7fe700 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 msgr2=0x7fdffc195a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-05-06T14:32:09.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 0x7fdffc195a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.153:47340/0 (socket says 172.21.15.153:47340) 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 -- 172.21.15.153:0/2969948975 learned_addr learned my addr 172.21.15.153:0/2969948975 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 -- 172.21.15.153:0/2969948975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 msgr2=0x7fdffc195a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 0x7fdffc195a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 -- 172.21.15.153:0/2969948975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec0095f0 con 0x7fdffc101350 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdffaffd700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc195f50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00bd70 tx=0x7fdfec00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfec01d070 con 0x7fdffc101350 2024-05-06T14:32:09.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.438+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc19b100 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.439+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdfec022470 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.439+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfec00f690 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.439+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc19b670 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.439+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc108d40 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.440+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 25) v1 ==== 91089+0+0 (secure 0 0 0) 0x7fdfec0225e0 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.440+0000 7fdff8ff9700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fdfe406d300 0x7fdfe406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.440+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fdfec08e3f0 con 0x7fdffc101350 2024-05-06T14:32:09.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.441+0000 7fdffb7fe700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fdfe406d300 0x7fdfe406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-06T14:32:09.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.441+0000 7fdffb7fe700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fdfe406d300 0x7fdfe406f7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdff0006fd0 tx=0x7fdff0006e00 comp rx=0 tx=0).ready entity=mgr.24373 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-06T14:32:09.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.445+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185428 (secure 0 0 0) 0x7fdfec05b700 con 0x7fdffc101350 2024-05-06T14:32:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.675+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 --> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdffc105ca0 con 0x7fdfe406d300 2024-05-06T14:32:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.676+0000 7fdff8ff9700 1 -- 172.21.15.153:0/2969948975 <== mgr.24373 v2:172.21.15.190:6828/998351904 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fdffc105ca0 con 0x7fdfe406d300 2024-05-06T14:32:09.664 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-05-06T14:32:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb", 2024-05-06T14:32:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-05-06T14:32:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T14:32:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-05-06T14:32:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "", 2024-05-06T14:32:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mgr daemons", 2024-05-06T14:32:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-05-06T14:32:09.665 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-05-06T14:32:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fdfe406d300 msgr2=0x7fdfe406f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fdfe406d300 0x7fdfe406f7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdff0006fd0 tx=0x7fdff0006e00 comp rx=0 tx=0).stop 2024-05-06T14:32:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 msgr2=0x7fdffc195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-06T14:32:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc195f50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00bd70 tx=0x7fdfec00be50 comp rx=0 tx=0).stop 2024-05-06T14:32:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 shutdown_connections 2024-05-06T14:32:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:6828/998351904,v1:172.21.15.190:6829/998351904] conn(0x7fdfe406d300 0x7fdfe406f7b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdffc100150 0x7fdffc195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 --2- 172.21.15.153:0/2969948975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdffc101350 0x7fdffc195f50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-06T14:32:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 >> 172.21.15.153:0/2969948975 conn(0x7fdffc0fb6e0 msgr2=0x7fdffc104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-06T14:32:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 shutdown_connections 2024-05-06T14:32:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-05-06T14:32:09.680+0000 7fe001bb2700 1 -- 172.21.15.153:0/2969948975 wait complete. 2024-05-06T14:32:10.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:10 smithi153 podman[70489]: 2024-05-06 14:32:10.285331256 +0000 UTC m=+2.553628510 container cleanup 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:32:10.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:10 smithi153 bash[70489]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153 2024-05-06T14:32:11.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:May 06 14:32:11 smithi153 podman[70602]: 2024-05-06 14:32:11.587201973 +0000 UTC m=+3.448811360 container remove 44f2993c90f456de0723b17c5fa8bbe09782f244ddacb8d36b44ed02d32c4337 (image=quay.io/ceph/ceph:v18.2.0, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi153, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-05-06T14:32:11.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi153.service' 2024-05-06T14:32:11.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:11.919 INFO:tasks.cephadm.mon.smithi153:Stopped mon.smithi153 2024-05-06T14:32:11.920 INFO:tasks.cephadm.mon.smithi190:Stopping mon.smithi190... 2024-05-06T14:32:11.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi190 2024-05-06T14:32:12.238 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:11 smithi190 systemd[1]: Stopping Ceph mon.smithi190 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:12.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:12 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi190[33657]: 2024-05-06T14:32:12.248+0000 7fd2865ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi190 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:12.685 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:12 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi190[33657]: 2024-05-06T14:32:12.248+0000 7fd2865ee700 -1 mon.smithi190@1(peon) e2 *** Got Signal Terminated *** 2024-05-06T14:32:12.686 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:12 smithi190 podman[58333]: 2024-05-06 14:32:12.422827034 +0000 UTC m=+0.284861657 container died 88aeab4dfa5eecc10e0280e2861e008ad860bbe5a7f93b52ab5db925afc425d8 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi190, GIT_BRANCH=HEAD, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:32:13.170 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:12 smithi190 podman[58333]: 2024-05-06 14:32:12.949772488 +0000 UTC m=+0.811807153 container cleanup 88aeab4dfa5eecc10e0280e2861e008ad860bbe5a7f93b52ab5db925afc425d8 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi190, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-05-06T14:32:13.170 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:12 smithi190 bash[58333]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi190 2024-05-06T14:32:13.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi190.service' 2024-05-06T14:32:13.428 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:13 smithi190 podman[58347]: 2024-05-06 14:32:13.184534286 +0000 UTC m=+0.759532255 container remove 88aeab4dfa5eecc10e0280e2861e008ad860bbe5a7f93b52ab5db925afc425d8 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-mon-smithi190, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD) 2024-05-06T14:32:13.429 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:13 smithi190 systemd[1]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi190.service: Deactivated successfully. 2024-05-06T14:32:13.429 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:13 smithi190 systemd[1]: Stopped Ceph mon.smithi190 for eacfcfb6-0bb3-11ef-bc97-c7b262605968. 2024-05-06T14:32:13.429 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:May 06 14:32:13 smithi190 systemd[1]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@mon.smithi190.service: Consumed 5.040s CPU time. 2024-05-06T14:32:13.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:13.455 INFO:tasks.cephadm.mon.smithi190:Stopped mon.smithi190 2024-05-06T14:32:13.455 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-06T14:32:13.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.0 2024-05-06T14:32:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:13 smithi153 systemd[1]: Stopping Ceph osd.0 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:15.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:15 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0[42069]: 2024-05-06T14:32:15.421+0000 7fc6a7d33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:15.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:15 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0[42069]: 2024-05-06T14:32:15.421+0000 7fc6a7d33700 -1 osd.0 49 *** Got signal Terminated *** 2024-05-06T14:32:15.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:15 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0[42069]: 2024-05-06T14:32:15.421+0000 7fc6a7d33700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:32:21.022 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:20 smithi153 podman[71187]: 2024-05-06 14:32:20.769971644 +0000 UTC m=+6.109420868 container died 6c202dfae018aa90c19942aefeeae256d3a663c495fb42bcd2987992f4b58e6b (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-06T14:32:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 podman[71187]: 2024-05-06 14:32:21.146490792 +0000 UTC m=+6.485940015 container cleanup 6c202dfae018aa90c19942aefeeae256d3a663c495fb42bcd2987992f4b58e6b (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:32:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 bash[71187]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0 2024-05-06T14:32:21.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 podman[71494]: 2024-05-06 14:32:21.411478976 +0000 UTC m=+0.641313378 container remove 6c202dfae018aa90c19942aefeeae256d3a663c495fb42bcd2987992f4b58e6b (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:32:22.017 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 podman[71587]: 2024-05-06 14:32:21.617656395 +0000 UTC m=+0.018193567 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-05-06T14:32:22.017 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 podman[71587]: 2024-05-06 14:32:21.737312976 +0000 UTC m=+0.137850132 container create c2daec98208ca649b4f226df9255b4acd362cbdc4250f241870f88dd2368cb37 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-06T14:32:22.018 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 podman[71587]: 2024-05-06 14:32:21.948522504 +0000 UTC m=+0.349059660 container init c2daec98208ca649b4f226df9255b4acd362cbdc4250f241870f88dd2368cb37 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0-deactivate, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-06T14:32:22.018 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:21 smithi153 podman[71587]: 2024-05-06 14:32:21.951945892 +0000 UTC m=+0.352483057 container start c2daec98208ca649b4f226df9255b4acd362cbdc4250f241870f88dd2368cb37 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0-deactivate, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-06T14:32:22.018 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:22 smithi153 podman[71587]: 2024-05-06 14:32:22.031681315 +0000 UTC m=+0.432218480 container attach c2daec98208ca649b4f226df9255b4acd362cbdc4250f241870f88dd2368cb37 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:32:22.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:May 06 14:32:22 smithi153 podman[71587]: 2024-05-06 14:32:22.242447904 +0000 UTC m=+0.642985057 container died c2daec98208ca649b4f226df9255b4acd362cbdc4250f241870f88dd2368cb37 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD) 2024-05-06T14:32:22.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.0.service' 2024-05-06T14:32:22.918 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:22.918 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-06T14:32:22.919 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-06T14:32:22.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.1 2024-05-06T14:32:23.241 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:23 smithi153 systemd[1]: Stopping Ceph osd.1 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:23 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1[47617]: 2024-05-06T14:32:23.254+0000 7fdf55945700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:23 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1[47617]: 2024-05-06T14:32:23.254+0000 7fdf55945700 -1 osd.1 49 *** Got signal Terminated *** 2024-05-06T14:32:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:23 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1[47617]: 2024-05-06T14:32:23.254+0000 7fdf55945700 -1 osd.1 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:32:28.842 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:28 smithi153 podman[71691]: 2024-05-06 14:32:28.589250318 +0000 UTC m=+5.436582049 container died 78ff3749996ccee3b8c4a6e566243b6755794bdd57c3d8abd9e03e8cffd19206 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T14:32:29.200 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:28 smithi153 podman[71691]: 2024-05-06 14:32:28.955306156 +0000 UTC m=+5.802637877 container cleanup 78ff3749996ccee3b8c4a6e566243b6755794bdd57c3d8abd9e03e8cffd19206 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T14:32:29.201 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:28 smithi153 bash[71691]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1 2024-05-06T14:32:29.514 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:29 smithi153 podman[71711]: 2024-05-06 14:32:29.214739475 +0000 UTC m=+0.616813075 container remove 78ff3749996ccee3b8c4a6e566243b6755794bdd57c3d8abd9e03e8cffd19206 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T14:32:29.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:29 smithi153 podman[71798]: 2024-05-06 14:32:29.431137883 +0000 UTC m=+0.017957150 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-05-06T14:32:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:29 smithi153 podman[71798]: 2024-05-06 14:32:29.557383287 +0000 UTC m=+0.144202586 container create 183fcf9a3714037e592cb540f23ef58b26a772114c11abfaa331ad915fb913b4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T14:32:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:29 smithi153 podman[71798]: 2024-05-06 14:32:29.759895072 +0000 UTC m=+0.346714325 container init 183fcf9a3714037e592cb540f23ef58b26a772114c11abfaa331ad915fb913b4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS) 2024-05-06T14:32:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:29 smithi153 podman[71798]: 2024-05-06 14:32:29.763021852 +0000 UTC m=+0.349841111 container start 183fcf9a3714037e592cb540f23ef58b26a772114c11abfaa331ad915fb913b4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1) 2024-05-06T14:32:30.035 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:29 smithi153 podman[71798]: 2024-05-06 14:32:29.843231443 +0000 UTC m=+0.430050711 container attach 183fcf9a3714037e592cb540f23ef58b26a772114c11abfaa331ad915fb913b4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS) 2024-05-06T14:32:30.310 INFO:journalctl@ceph.osd.1.smithi153.stdout:May 06 14:32:30 smithi153 podman[71798]: 2024-05-06 14:32:30.049545509 +0000 UTC m=+0.636364765 container died 183fcf9a3714037e592cb540f23ef58b26a772114c11abfaa331ad915fb913b4 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD) 2024-05-06T14:32:30.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.1.service' 2024-05-06T14:32:30.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:30.696 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-06T14:32:30.696 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-06T14:32:30.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.2 2024-05-06T14:32:31.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:30 smithi153 systemd[1]: Stopping Ceph osd.2 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:31 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2[53091]: 2024-05-06T14:32:31.041+0000 7f51e1710700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:31 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2[53091]: 2024-05-06T14:32:31.041+0000 7f51e1710700 -1 osd.2 49 *** Got signal Terminated *** 2024-05-06T14:32:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:31 smithi153 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2[53091]: 2024-05-06T14:32:31.041+0000 7f51e1710700 -1 osd.2 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:32:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:36 smithi153 podman[71912]: 2024-05-06 14:32:36.534371318 +0000 UTC m=+5.594829627 container died 3fbd486b6ac674d57cb23a75e5efb9885be23334ef320410b30187eceb232b22 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:32:37.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:36 smithi153 podman[71912]: 2024-05-06 14:32:36.865869283 +0000 UTC m=+5.926327584 container cleanup 3fbd486b6ac674d57cb23a75e5efb9885be23334ef320410b30187eceb232b22 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-05-06T14:32:37.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:36 smithi153 bash[71912]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2 2024-05-06T14:32:37.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[71932]: 2024-05-06 14:32:37.08378797 +0000 UTC m=+0.544631512 container remove 3fbd486b6ac674d57cb23a75e5efb9885be23334ef320410b30187eceb232b22 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:32:37.715 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[72023]: 2024-05-06 14:32:37.316604511 +0000 UTC m=+0.018643477 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-05-06T14:32:37.715 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[72023]: 2024-05-06 14:32:37.417974958 +0000 UTC m=+0.120013912 container create d6138624fc30d62927bc73eebb4f7cb4c5b7a2fa79e78cab9327f81693c67235 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-06T14:32:37.715 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[72023]: 2024-05-06 14:32:37.629052242 +0000 UTC m=+0.331091195 container init d6138624fc30d62927bc73eebb4f7cb4c5b7a2fa79e78cab9327f81693c67235 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:32:37.715 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[72023]: 2024-05-06 14:32:37.63213563 +0000 UTC m=+0.334174583 container start d6138624fc30d62927bc73eebb4f7cb4c5b7a2fa79e78cab9327f81693c67235 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=-18.2.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:32:37.715 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[72023]: 2024-05-06 14:32:37.728948333 +0000 UTC m=+0.430987298 container attach d6138624fc30d62927bc73eebb4f7cb4c5b7a2fa79e78cab9327f81693c67235 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:32:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:37 smithi153 podman[72023]: 2024-05-06 14:32:37.924352107 +0000 UTC m=+0.626391061 container died d6138624fc30d62927bc73eebb4f7cb4c5b7a2fa79e78cab9327f81693c67235 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20231212) 2024-05-06T14:32:38.504 INFO:journalctl@ceph.osd.2.smithi153.stdout:May 06 14:32:38 smithi153 podman[72049]: 2024-05-06 14:32:38.515555193 +0000 UTC m=+0.583995352 container remove d6138624fc30d62927bc73eebb4f7cb4c5b7a2fa79e78cab9327f81693c67235 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212) 2024-05-06T14:32:38.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.2.service' 2024-05-06T14:32:38.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:38.559 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-06T14:32:38.559 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-06T14:32:38.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.3 2024-05-06T14:32:38.855 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:38 smithi190 systemd[1]: Stopping Ceph osd.3 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:39.185 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:38 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:32:38.869+0000 7fd3e3dec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:39.185 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:38 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:32:38.869+0000 7fd3e3dec700 -1 osd.3 49 *** Got signal Terminated *** 2024-05-06T14:32:39.185 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:38 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:32:38.869+0000 7fd3e3dec700 -1 osd.3 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:32:42.435 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:42 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:32:42.056+0000 7fd3e0be3700 -1 osd.3 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.208084+0000 front 2024-05-06T14:32:18.208106+0000 (oldest deadline 2024-05-06T14:32:41.107998+0000) 2024-05-06T14:32:43.435 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:43 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3[39625]: 2024-05-06T14:32:43.025+0000 7fd3e0be3700 -1 osd.3 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.208084+0000 front 2024-05-06T14:32:18.208106+0000 (oldest deadline 2024-05-06T14:32:41.107998+0000) 2024-05-06T14:32:43.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:43 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:43.013+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:43.935 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:43 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:43.661+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:44.340 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:44 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:44.033+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:44.653 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:44 smithi190 podman[58465]: 2024-05-06 14:32:44.353179696 +0000 UTC m=+5.585213477 container died 79af272339eb72b7bfc93313b514319c79e3f4e3a4b36881e24d734a3396c982 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.build-date=20231212, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-06T14:32:44.935 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:44 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:44.666+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:44.935 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:44 smithi190 podman[58465]: 2024-05-06 14:32:44.85318766 +0000 UTC m=+6.085221447 container cleanup 79af272339eb72b7bfc93313b514319c79e3f4e3a4b36881e24d734a3396c982 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-06T14:32:44.935 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:44 smithi190 bash[58465]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3 2024-05-06T14:32:45.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:45 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:45.049+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:45.363 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58482]: 2024-05-06 14:32:45.071443206 +0000 UTC m=+0.714859064 container remove 79af272339eb72b7bfc93313b514319c79e3f4e3a4b36881e24d734a3396c982 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20231212) 2024-05-06T14:32:45.678 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58569]: 2024-05-06 14:32:45.279806609 +0000 UTC m=+0.018501593 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-05-06T14:32:45.678 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58569]: 2024-05-06 14:32:45.397685175 +0000 UTC m=+0.136380194 container create ad2306a5c68539a8b4d2ff950471fe857f27a516233bbb609d184104bb3f6649 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3-deactivate, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212) 2024-05-06T14:32:45.678 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58569]: 2024-05-06 14:32:45.609259035 +0000 UTC m=+0.347954023 container init ad2306a5c68539a8b4d2ff950471fe857f27a516233bbb609d184104bb3f6649 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:32:45.678 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58569]: 2024-05-06 14:32:45.612694761 +0000 UTC m=+0.351389738 container start ad2306a5c68539a8b4d2ff950471fe857f27a516233bbb609d184104bb3f6649 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:32:45.679 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58569]: 2024-05-06 14:32:45.692024859 +0000 UTC m=+0.430719855 container attach ad2306a5c68539a8b4d2ff950471fe857f27a516233bbb609d184104bb3f6649 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD) 2024-05-06T14:32:45.935 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:45 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:45.711+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:45.935 INFO:journalctl@ceph.osd.3.smithi190.stdout:May 06 14:32:45 smithi190 podman[58569]: 2024-05-06 14:32:45.899336298 +0000 UTC m=+0.638031267 container died ad2306a5c68539a8b4d2ff950471fe857f27a516233bbb609d184104bb3f6649 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-06T14:32:46.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:46 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:46.009+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:46.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.3.service' 2024-05-06T14:32:46.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:46.676 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-06T14:32:46.676 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-06T14:32:46.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.4 2024-05-06T14:32:46.900 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:46 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:46.695+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:46.900 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:46 smithi190 systemd[1]: Stopping Ceph osd.4 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:47.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:47 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:47.032+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:47.186 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:47 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:47.032+0000 7fd5e42fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:47.186 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:47 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:47.032+0000 7fd5e42fa700 -1 osd.4 49 *** Got signal Terminated *** 2024-05-06T14:32:47.186 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:47 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:47.032+0000 7fd5e42fa700 -1 osd.4 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:32:48.053 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:47 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:47.744+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:48.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:48 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:48.065+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:49.032 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:48 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:48.722+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:49.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:49 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:49.045+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:50.005 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:49 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:49.736+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:50.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:50 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:50.018+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:51.018 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:50 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:50.779+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:51.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:51 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:51.031+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:52.185 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:51 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:51.825+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:18.225304+0000 front 2024-05-06T14:32:18.225267+0000 (oldest deadline 2024-05-06T14:32:42.925300+0000) 2024-05-06T14:32:52.185 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:51 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4[44294]: 2024-05-06T14:32:51.825+0000 7fd5dd0e9700 -1 osd.4 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:27.626114+0000 front 2024-05-06T14:32:27.626156+0000 (oldest deadline 2024-05-06T14:32:51.726107+0000) 2024-05-06T14:32:52.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:51 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:51.988+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:52.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:51 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:51.988+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:52.685 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:52 smithi190 podman[58674]: 2024-05-06 14:32:52.448269232 +0000 UTC m=+5.538573121 container died 6db95b0197f1eb1434f85dc756508c4c1936fa1c17292fb3d26d03a947a2966e (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-05-06T14:32:53.185 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:52 smithi190 podman[58674]: 2024-05-06 14:32:52.942454576 +0000 UTC m=+6.032758474 container cleanup 6db95b0197f1eb1434f85dc756508c4c1936fa1c17292fb3d26d03a947a2966e (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212) 2024-05-06T14:32:53.185 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:52 smithi190 bash[58674]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4 2024-05-06T14:32:53.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:52 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:52.994+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:53.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:52 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:52.994+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:53.482 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:53 smithi190 podman[58703]: 2024-05-06 14:32:53.202258843 +0000 UTC m=+0.745529414 container remove 6db95b0197f1eb1434f85dc756508c4c1936fa1c17292fb3d26d03a947a2966e (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212) 2024-05-06T14:32:53.783 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:53 smithi190 podman[58792]: 2024-05-06 14:32:53.399856695 +0000 UTC m=+0.018405668 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-05-06T14:32:53.783 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:53 smithi190 podman[58792]: 2024-05-06 14:32:53.503255874 +0000 UTC m=+0.121804830 container create 036ebd830631aa0b24c7cce75e1a085b06c596804467741af6c3d880fec2b8ef (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, ceph=True, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1) 2024-05-06T14:32:53.784 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:53 smithi190 podman[58792]: 2024-05-06 14:32:53.714231009 +0000 UTC m=+0.332779971 container init 036ebd830631aa0b24c7cce75e1a085b06c596804467741af6c3d880fec2b8ef (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4-deactivate, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.0, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-05-06T14:32:53.784 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:53 smithi190 podman[58792]: 2024-05-06 14:32:53.717276897 +0000 UTC m=+0.335825859 container start 036ebd830631aa0b24c7cce75e1a085b06c596804467741af6c3d880fec2b8ef (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1) 2024-05-06T14:32:53.784 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:53 smithi190 podman[58792]: 2024-05-06 14:32:53.797405653 +0000 UTC m=+0.415954604 container attach 036ebd830631aa0b24c7cce75e1a085b06c596804467741af6c3d880fec2b8ef (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2024-05-06T14:32:54.184 INFO:journalctl@ceph.osd.4.smithi190.stdout:May 06 14:32:54 smithi190 podman[58792]: 2024-05-06 14:32:54.007022678 +0000 UTC m=+0.625571640 container died 036ebd830631aa0b24c7cce75e1a085b06c596804467741af6c3d880fec2b8ef (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:32:54.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:53 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:53.985+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:54.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:53 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:53.985+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:54.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.4.service' 2024-05-06T14:32:54.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:32:54.721 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-06T14:32:54.722 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-06T14:32:54.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.5 2024-05-06T14:32:54.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:54 smithi190 systemd[1]: Stopping Ceph osd.5 for eacfcfb6-0bb3-11ef-bc97-c7b262605968... 2024-05-06T14:32:55.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.001+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:55.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.001+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:55.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.078+0000 7f157b5d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:32:55.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.078+0000 7f157b5d4700 -1 osd.5 49 *** Got signal Terminated *** 2024-05-06T14:32:55.436 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.078+0000 7f157b5d4700 -1 osd.5 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:32:56.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.983+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:56.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:55 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:55.983+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:57.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:57 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:57.001+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:57.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:57 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:57.001+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:58.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:58 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:58.044+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:58.436 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:58 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:58.044+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:58.436 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:58 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:58.044+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-05-06T14:32:35.925361+0000 front 2024-05-06T14:32:35.925406+0000 (oldest deadline 2024-05-06T14:32:57.025199+0000) 2024-05-06T14:32:59.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:59 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:59.069+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-05-06T14:32:16.722794+0000 front 2024-05-06T14:32:16.722803+0000 (oldest deadline 2024-05-06T14:32:42.022687+0000) 2024-05-06T14:32:59.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:59 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:59.069+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-05-06T14:32:26.024041+0000 front 2024-05-06T14:32:26.024090+0000 (oldest deadline 2024-05-06T14:32:51.923835+0000) 2024-05-06T14:32:59.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:32:59 smithi190 ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5[48973]: 2024-05-06T14:32:59.069+0000 7f1576bc8700 -1 osd.5 49 heartbeat_check: no reply from 172.21.15.153:6822 osd.2 since back 2024-05-06T14:32:35.925361+0000 front 2024-05-06T14:32:35.925406+0000 (oldest deadline 2024-05-06T14:32:57.025199+0000) 2024-05-06T14:33:00.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:00 smithi190 podman[58898]: 2024-05-06 14:33:00.585998182 +0000 UTC m=+5.629768851 container died ae3651b2db8d1bc4718c6a3cfd7a211a25e0075ff2575ca1e70d933d605ebdc2 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:33:01.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[58898]: 2024-05-06 14:33:01.062909136 +0000 UTC m=+6.106679806 container cleanup ae3651b2db8d1bc4718c6a3cfd7a211a25e0075ff2575ca1e70d933d605ebdc2 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=-18.2.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True) 2024-05-06T14:33:01.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 bash[58898]: ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5 2024-05-06T14:33:01.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[58916]: 2024-05-06 14:33:01.297894639 +0000 UTC m=+0.706936429 container remove ae3651b2db8d1bc4718c6a3cfd7a211a25e0075ff2575ca1e70d933d605ebdc2 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5, CEPH_POINT_RELEASE=-18.2.0, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-06T14:33:01.832 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[59007]: 2024-05-06 14:33:01.497952923 +0000 UTC m=+0.018390955 image pull dc2bc1663786ad5862ef6efcff18243ab7dfeda0bcb911ddf757a0d20feca946 quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007 2024-05-06T14:33:01.832 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[59007]: 2024-05-06 14:33:01.615488727 +0000 UTC m=+0.135926744 container create 29408cdf616bd86f6a5ddad505aac8e636fb3c789a010d12b0b3511de8be5054 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5-deactivate, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0) 2024-05-06T14:33:01.832 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[59007]: 2024-05-06 14:33:01.842895291 +0000 UTC m=+0.363333319 container init 29408cdf616bd86f6a5ddad505aac8e636fb3c789a010d12b0b3511de8be5054 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20231212, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0) 2024-05-06T14:33:01.832 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[59007]: 2024-05-06 14:33:01.846189317 +0000 UTC m=+0.366627337 container start 29408cdf616bd86f6a5ddad505aac8e636fb3c789a010d12b0b3511de8be5054 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-06T14:33:02.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:01 smithi190 podman[59007]: 2024-05-06 14:33:01.934782501 +0000 UTC m=+0.455220532 container attach 29408cdf616bd86f6a5ddad505aac8e636fb3c789a010d12b0b3511de8be5054 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-06T14:33:02.435 INFO:journalctl@ceph.osd.5.smithi190.stdout:May 06 14:33:02 smithi190 podman[59007]: 2024-05-06 14:33:02.125137716 +0000 UTC m=+0.645575738 container died 29408cdf616bd86f6a5ddad505aac8e636fb3c789a010d12b0b3511de8be5054 (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.vendor=CentOS) 2024-05-06T14:33:02.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eacfcfb6-0bb3-11ef-bc97-c7b262605968@osd.5.service' 2024-05-06T14:33:02.731 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:33:02.732 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-06T14:33:02.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-06T14:33:02.888 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:33:21.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-06T14:33:21.709 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:33:36.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:33:36.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:33:37.009 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-06T14:33:37.010 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/remote/smithi153/crash 2024-05-06T14:33:37.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/crash -- . 2024-05-06T14:33:37.046 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-06T14:33:37.046 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-05-06T14:33:37.048 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/remote/smithi190/crash 2024-05-06T14:33:37.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/crash -- . 2024-05-06T14:33:37.071 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-06T14:33:37.071 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-05-06T14:33:37.072 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-06T14:33:37.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-05-06T14:33:37.124 INFO:tasks.cephadm:Compressing logs... 2024-05-06T14:33:37.124 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T14:33:37.166 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T14:33:37.188 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-06T14:33:37.188 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-05-06T14:33:37.188 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log 2024-05-06T14:33:37.188 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi153.log 2024-05-06T14:33:37.189 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-06T14:33:37.189 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-06T14:33:37.195 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mgr.smithi153.dolfed.log 2024-05-06T14:33:37.195 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi153.log: /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.2.log: /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-06T14:33:37.195 INFO:teuthology.orchestra.run.smithi153.stderr: 87.1%/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mgr.smithi153.dolfed.log: -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr: 93.7% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi153.log.gz 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi153.hiwroq.log 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.1.log: /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi153.hiwroq.log: 90.7% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.cephadm.log: 84.2% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-06T14:33:37.196 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi153.ftbnnk.log 2024-05-06T14:33:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-06T14:33:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2024-05-06T14:33:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log 2024-05-06T14:33:37.199 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi190.log 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mon.smithi190.log 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi190.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.5.log 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr: 87.0% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mon.smithi190.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.3.log 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-client.ceph-exporter.smithi190.log.gz 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:gzip/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.5.log: -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.4.log 2024-05-06T14:33:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi190.psnzbz.log 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr: 90.8% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mgr.smithi190.qauohf.log 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi190.psnzbz.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mgr.smithi190.qauohf.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi190.quxdec.log 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr: 83.3% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-06T14:33:37.201 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi190.quxdec.log: /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-volume.log: 93.9% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi190.quxdec.log.gz 2024-05-06T14:33:37.202 INFO:teuthology.orchestra.run.smithi190.stderr: 91.7% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-06T14:33:37.203 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-volume.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-06T14:33:37.203 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mon.smithi153.log 2024-05-06T14:33:37.203 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi153.ftbnnk.log: 91.5% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-06T14:33:37.204 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mon.smithi153.log: 90.7% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi153.hiwroq.log.gz 2024-05-06T14:33:37.237 INFO:teuthology.orchestra.run.smithi190.stderr: 89.9% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mgr.smithi190.qauohf.log.gz 2024-05-06T14:33:37.303 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mon.smithi190.log.gz 2024-05-06T14:33:37.334 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mgr.smithi153.dolfed.log.gz 2024-05-06T14:33:37.498 INFO:teuthology.orchestra.run.smithi153.stderr: 90.9% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mon.smithi153.log.gz 2024-05-06T14:33:38.425 INFO:teuthology.orchestra.run.smithi190.stderr: 93.7% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.4.log.gz 2024-05-06T14:33:38.453 INFO:teuthology.orchestra.run.smithi153.stderr: 93.9% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-06T14:33:38.718 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-06T14:33:38.829 INFO:teuthology.orchestra.run.smithi153.stderr: 93.8% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-06T14:33:38.935 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.3.log.gz 2024-05-06T14:33:38.970 INFO:teuthology.orchestra.run.smithi190.stderr: 93.7% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-osd.5.log.gz 2024-05-06T14:33:41.487 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi190.psnzbz.log.gz 2024-05-06T14:33:41.488 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-05-06T14:33:41.488 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m4.308s 2024-05-06T14:33:41.488 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m8.908s 2024-05-06T14:33:41.488 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.325s 2024-05-06T14:34:02.617 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/eacfcfb6-0bb3-11ef-bc97-c7b262605968/ceph-mds.cephfs.smithi153.ftbnnk.log.gz 2024-05-06T14:34:02.619 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-05-06T14:34:02.619 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m25.440s 2024-05-06T14:34:02.619 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m29.318s 2024-05-06T14:34:02.619 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.976s 2024-05-06T14:34:02.619 INFO:tasks.cephadm:Archiving logs... 2024-05-06T14:34:02.620 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/remote/smithi153/log 2024-05-06T14:34:02.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T14:34:07.480 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/remote/smithi190/log 2024-05-06T14:34:07.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T14:34:08.877 INFO:tasks.cephadm:Removing cluster... 2024-05-06T14:34:08.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 --force 2024-05-06T14:34:09.031 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:34:09.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eacfcfb6-0bb3-11ef-bc97-c7b262605968 --force 2024-05-06T14:34:09.639 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: eacfcfb6-0bb3-11ef-bc97-c7b262605968 2024-05-06T14:34:09.911 INFO:tasks.cephadm:Removing cephadm ... 2024-05-06T14:34:09.912 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-06T14:34:09.928 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-06T14:34:09.943 INFO:tasks.cephadm:Teardown complete 2024-05-06T14:34:09.943 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-06T14:34:09.956 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-06T14:34:09.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-06T14:34:09.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-06T14:34:10.058 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-06T14:34:10.076 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-06T14:34:10.076 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T14:34:10.079 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T14:34:10.091 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-05-06T14:34:10.093 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-05-06T14:34:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T14:34:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-05-06T14:34:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 -364us[ -373us] +/- 60ms 2024-05-06T14:34:10.095 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -2628us[-2637us] +/- 72ms 2024-05-06T14:34:10.095 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +2973us[+2965us] +/- 68ms 2024-05-06T14:34:10.095 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:34:10.096 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T14:34:10.097 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-05-06T14:34:10.097 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 -410us[ -413us] +/- 60ms 2024-05-06T14:34:10.097 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -2687us[-2690us] +/- 72ms 2024-05-06T14:34:10.097 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +2929us[+2929us] +/- 68ms 2024-05-06T14:34:10.097 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:34:10.098 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-06T14:34:10.110 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-06T14:34:10.111 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-06T14:34:10.122 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T14:34:10.206 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T14:34:10.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-06T14:34:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:34:10.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-06T14:34:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:34:10.324 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-06T14:34:10.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-06T14:34:10.347 INFO:teuthology.task.internal:Duration was 1131.767246 seconds 2024-05-06T14:34:10.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-06T14:34:10.358 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-06T14:34:10.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T14:34:10.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T14:34:10.395 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T14:34:10.399 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T14:34:10.822 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-06T14:34:10.822 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-05-06T14:34:10.822 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T14:34:10.881 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-05-06T14:34:10.882 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T14:34:10.910 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-06T14:34:10.911 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T14:34:10.924 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T14:34:10.975 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-06T14:34:10.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T14:34:10.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T14:34:11.341 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-06T14:34:11.352 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-06T14:34:11.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T14:34:11.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T14:34:11.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-06T14:34:11.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:34:11.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:34:11.438 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-05-06T14:34:11.460 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-05-06T14:34:11.473 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:34:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:34:11.504 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:34:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:34:11.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-06T14:34:11.538 INFO:teuthology.task.internal:Transferring archived files... 2024-05-06T14:34:11.539 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/remote/smithi153 2024-05-06T14:34:11.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T14:34:11.583 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254/remote/smithi190 2024-05-06T14:34:11.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T14:34:11.616 INFO:teuthology.task.internal:Removing archive directory... 2024-05-06T14:34:11.617 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T14:34:11.619 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T14:34:11.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-06T14:34:11.679 INFO:teuthology.task.internal:Not uploading archives. 2024-05-06T14:34:11.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-06T14:34:11.691 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-06T14:34:11.692 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T14:34:11.694 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T14:34:11.706 INFO:teuthology.orchestra.run.smithi153.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 14:34 /home/ubuntu/cephtest 2024-05-06T14:34:11.720 INFO:teuthology.orchestra.run.smithi190.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 14:34 /home/ubuntu/cephtest 2024-05-06T14:34:11.721 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-06T14:34:11.734 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-06T14:34:11.797 INFO:teuthology.nuke:Checking targets against current locks 2024-05-06T14:34:11.821 DEBUG:teuthology.nuke:shortname: smithi153 2024-05-06T14:34:11.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T14:34:11.843 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 14:07:30.112023', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNDhQqShhO5NXWOND1UifgILZQ1beRwhLV9L73gNkq1ZKTMDxl1bv6nRVSwT83u386wiZyUy7YCMTPc1oy2BWM='} 2024-05-06T14:34:11.847 DEBUG:teuthology.nuke:shortname: smithi190 2024-05-06T14:34:11.847 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T14:34:11.870 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 14:07:30.110001', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6o9s/jwV00BMmczw71GWwoGiRRXXko1PMNfztbEN9AuXPcG6uVIS6734lMITpMwqZj7tL/2ZowTGWJ7f6WL/M='} 2024-05-06T14:34:11.901 INFO:teuthology.orchestra.console.smithi153:Power off 2024-05-06T14:34:11.902 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T14:34:11.928 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-05-06T14:34:11.929 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:11.944 INFO:teuthology.orchestra.console.smithi190:Power off 2024-05-06T14:34:11.944 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T14:34:11.957 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-05-06T14:34:11.967 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-05-06T14:34:11.968 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:11.995 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-05-06T14:34:15.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:34:15.958 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:15.996 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:16.069 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-05-06T14:34:16.107 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-05-06T14:34:20.074 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:20.108 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:20.186 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-05-06T14:34:20.219 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-05-06T14:34:20.319 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-05-06T14:34:24.190 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:34:24.301 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-05-06T14:34:24.401 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-05-06T14:34:24.507 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1131.7672455310822 failure_reason: 'Command failed (workunit test suites/fsstress.sh) on smithi153 with status 126: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=70ce9539b62a15b9de429c1f0e91a1e012b099aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6fcda01840dc42c09fa3d59355acdcff status: fail success: false 2024-05-06T14:34:24.508 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T14:34:24.599 INFO:teuthology.run:FAIL